parsing large compressed xml files, python

Question

file  = BZ2File(SOME_FILE_PATH)
p = xml.parsers.expat.ParserCreate()
p.Parse(file)

Here's code that tries to parse xml file compressed with bz2. Unfortunately it fails with a message:

TypeError: Parse() argument 1 must be string or read-only buffer, not bz2.BZ2File

Is there a way to parse on the fly compressed bz2 xml files?

Note: p.Parse(file.read()) is not an option here. I want to parse a file which is larger than available memory, so I need to have a stream.

Answer 1

Use .read() on the file object to read in the entire file as a string, and then pass that to Parse?

file  = BZ2File(SOME_FILE_PATH)
p = xml.parsers.expat.ParserCreate()
p.Parse(file.read())

Answer 2

Can you pass in an mmap()'ed file? That should take care of automatically paging the needed parts of the file in, and avoid memory overflow. Of course if expat builts a parse tree, it might still run out of memory.

http://docs.python.org/library/mmap.html

Memory-mapped file objects behave like both strings and like file objects. Unlike normal string objects, however, these are mutable. You can use mmap objects in most places where strings are expected; for example, you can use the re module to search through a memory-mapped file.

Answer 3

Just use p.ParseFile(file) instead of p.Parse(file).

Parse() takes a string, ParseFile() takes a file handle, and reads the data in as required.

Ref: http://docs.python.org/library/pyexpat.html#xml.parsers.expat.xmlparser.ParseFile