Java's Scanner vs String.split() vs StringTokenizer; which should I use?

Question

I am currently using split() to scan through a file where each line has number of strings delimited by '~'. I read somewhere that Scanner could do a better job with a long file, performance-wise, so I thought about checking it out.

My question is: Would I have to create two instances of Scanner? That is, one to read a line and another one based on the line to get tokens for a delimiter? If I have to do so, I doubt if I would get any advantage from using it. Maybe I am missing something here?

Answer 1

I would say split() is fastest, and probably good enough for what you're doing. It is less flexible than scanner though. StringTokenizer is deprecated and is only available for backwards compatibility, so don't use it.

EDIT: You could always test both implementations to see which one is faster. I'm curious myself if scanner could be faster than split(). Split might be faster for a given size vs Scanner, but I can't be certain of that.

Answer 2

For processing line you can use scanner and for getting tokens from each line you can use split.

Scanner scanner = new Scanner(new File(loc));
try {
    while ( scanner.hasNextLine() ){
        String[] tokens = scanner.nextLine().split("~");
        // do the processing for tokens here
    }
}
finally {
    scanner.close();
}

Answer 3

You can use the useDelimiter("~") method to let you iterate through the tokens on each line with hasNext()/next(), while still using hasNextLine()/nextLine() to iterate through the lines themselves.

EDIT: If you're going to do a performance comparison, you should pre-compile the regex when you do the split() test:

Pattern splitRegex = Pattern.compile("~");
while ((line = bufferedReader.readLine()) != null)
{
  String[] tokens = splitRegex.split(line);
  // etc.
}

If you use String#split(String regex), the regex will be recompiled every time. (Scanner automatically caches all regexes the first time it compiles them.) If you do that, I wouldn't expect to see much difference in performance.