How to get the next token (int, float or string) from a file in Python?

Question

Is there some way to just get the next token from a file in Python, as for example the Scanner class does in Java?

File file = new File("something");
Scanner in = new Scanner(file);
double a = in.nextDouble();
String s = in.next();

I'd like to ignore whitespaces, tabs, newlines and just get the next int/float/word from the file. I know I could read the lines and build something like Scanner myself, but I'd like to know if there isn't already something that I could use.

I've searched around but could only find line-oriented methods.

Thank you!

Answer 1

if your file is *.ini alike text files, you could use ConfigParser module

There is few examples out there.

http://docs.python.org/library/configparser.html

and pyparsing will do that for other purpose, I think. I havn't use pyparsing before, so I have no clue right now.

http://pyparsing.wikispaces.com/

Answer 2

Probably you can take a look at PLY

Answer 3

Check out the shlex-module in the standard library: http://docs.python.org/library/shlex.html

import shlex
import StringIO # use in place of files

list(shlex.shlex(StringIO.StringIO('Some tokens. 123, 45.67 "A string with whitespace"')))

It does not handle floats the way you seem to want. Maybe you can extend or modify it.

Answer 4

I don't think there is really something around that sophisticated.

But you can take a look at the following options

• use re.split(pattern, string) and get what you want by providing regex's
• There is somewhere a Scanner class in the re module (but I don't think they developed it further)
• You could also consider using tokenize + StringIO
• Or as you yourself mentioned: Build one yourself, donate it do community and get famous ;)