Decompressing a .bz2 file in Python

Question

So, this is a seemingly simple question, but I'm apparently very very dull. I have a little script that downloads all the .bz2 files from a webpage, but for some reason the decompressing of that file is giving me a MAJOR headache.

I'm quite a Python newbie, so the answer is probably quite obvious, please help me.

In this bit of the script, I already have the file, and I just want to read it out to a variable, then decompress that? Is that right? I've tried all sorts of way to do this, I usually get "ValueError: couldn't find end of stream" error on the last line in this snippet. I've tried to open up the zipfile and write it out to a string in a zillion different ways. This is the latest.

        openZip = open(zipFile, "r")

        s = ''

        while True:
            newLine = openZip.readline()
            if(len(newLine)==0):
                break
            s+=newLine

        print s

        uncompressedData = bz2.decompress(s)

Hi Alex, I should've listed all the other methods I've tried, as I've tried the read() way.

METHOD A: print 'decompressing ' + filename

              fileHandle = open(zipFile)
              uncompressedData = ''

              while True:            
                    s = fileHandle.read(1024)
                    if not s:
                      break
                    print('RAW "%s"', s)
                    uncompressedData += bz2.decompress(s)

              uncompressedData += bz2.flush()

              newFile = open(steamTF2mapdir + filename.split(".bz2")[0],"w")
              newFile.write(uncompressedData)
              newFile.close()

I get the error :
uncompressedData += bz2.decompress(s) ValueError: couldn't find end of stream

METHOD B zipFile = steamTF2mapdir + filename print 'decompressing ' + filename fileHandle = open(zipFile)

        s = fileHandle.read()

        uncompressedData = bz2.decompress(s)

Same error : uncompressedData = bz2.decompress(s) ValueError: couldn't find end of stream

Thanks so much for you prompt reply. I'm really banging my head against the wall, feeling inordinately thick for not being able to decompress a simple .bz2 file.

By the by, used 7zip to decompress it manually, to make sure the file isn't wonky or anything, and it decompresses fine.

Answer 1

You're opening and reading the compressed file as if it was a textfile made up of lines. DON'T! It's NOT.

uncompressedData = bz2.BZ2File(zipFile).read()

seems to be closer to what you're angling for.

Edit: the OP has shown a few more things he's tried (though I don't see any notes about having tried the best method -- the one-liner I recommend above!) but they seem to all have one error in common, and I repeat the key bits from above:

opening ... the compressed file as if it was a textfile ... It's NOT.

open(filename) and even the more explicit open(filename, 'r') open, for reading, a text file -- a compressed file is a binary file, so in order to read it correctly you must open it with open(filename, 'rb'). ((my recommended bz2.BZ2File KNOWS it's dealing with a compressed file, of course, so there's no need to tell it anything more)).

In Python 2.*, on Unix-y systems (i.e. every system except Windows), you could get away with a sloppy use of open (but in Python 3.* you can't, as text is Unicode, while binary is bytes -- different types).

In Windows (and before then in DOS) it's always been indispensable to distinguish, as Windows' text files, for historical reason, are peculiar (use two bytes rather than one to end lines, and, at least in some cases, take a byte worth '\0x1A' as meaning a logical end of file) and so the reading and writing low-level code must compensate.

So I suspect the OP is using Windows and is paying the price for not carefully using the 'rb' option ("read binary") to the open built-in. (though bz2.BZ2File is still simpler, whatever platform you're using!-).

Answer 2

openZip = open(zipFile, "r")

If you're running on Windows, you may want to do say openZip = open(zipFile, "rb") here since the file is likely to contain CR/LF combinations, and you don't want them to be translated.

newLine = openZip.readline()

As Alex pointed out, this is very wrong, as the concept of "lines" is foreign to a compressed stream.

s = fileHandle.read(1024) [...] uncompressedData += bz2.decompress(s)

This is wrong for the same reason. 1024-byte chunks aren't likely to mean much to the decompressor, since it's going to want to work with it's own block-size.

s = fileHandle.read() uncompressedData = bz2.decompress(s)

If that doesn't work, I'd say it's the new-line translation problem I mentioned above.

Answer 3

This was very helpful. 44 of 2300 files gave an end of file missing error, on Windows open. Adding the b(inary) flag to open fixed the problem.

for line in bz2.BZ2File(filename, 'rb', 10000000) :

works well. (the 10M is the buffering size that works well with the large files involved)

Thanks!