How long should it take to stream a 1GB file in python on say a 2Ghz Intel Core 2 Duo machine?
fp = open('publisher_feed_8663.xml')
for line in fp:
a = line.split('<')
I suppose I wasn't specific enough. This process takes 20+ minutes which is abnormally long. Based on empirical data, what is a reasonable time?
That will entirely depend on what's in the file. You're reading it a line at a time, which will mean a load of overhead calling the iterator again and again for the common case of lots of short lines. Use fp.read(CHUNK) with some large number for CHUNK to improve performance.
However, I'm not sure what you're doing with split('<'). You can't usefully process XML with tools as basic as that, or with line-at-a-time parsing, since XML is not line-based. If you actually want to do something with the XML infoset in the file as you read it, you should consider a SAX parser. (Then again, 1GB of XML? That's already non-sensible really.)
Other people have talked about the time, I'll talk about the processing (XML aside).
If you're doing something this massive, you should certainly look at generators. This pdf will teach you basically all you will ever need to know about generators. Any time you are either consuming or producing large amounts of data (especially serially) generators should be your very best friend.
Your answer:
start = time.time()
fp = open('publisher_feed_8663.xml')
for line in fp:
a = line.split('<')
print time.time() - start
You will require a 1GB file named publisher_feed_8663.xml, python and a 2Ghz Intel Core 2 Duo machine.
For parsing of XML, you probably want to use an event based stream parser, such as SAX or lxml. I recommend reading the lxml documentation about iterparse: http://codespeak.net/lxml/parsing.html#iterparse-and-iterwalk
As for how long should this take, you can do trivial harddrive benchmarks on linux using tools like hdparm -tT /dev/sda.
More RAM always helps with processing large files, as the OS can keep a bigger disk cache.