Writing binary data to middle of a sparse file in Python

Question

I need to compile a binary file in pieces with pieces arriving in random order (yes, its a P2P project)

def write(filename, offset, data) 
    file.open(filename, "ab")
    file.seek(offset) 
    file.write(data) 
    file.close()

Say I have a 32KB write(f, o, d) at offset 1MB into file and then another 32KB write(f, o, d) at offset 0

I end up with a file 65KB in length (i.e. the gap consisting of 0s between 32KB - 1MB is truncated/disappears)

I am aware this may appear an incredibly stupid question, but I cannot seem to figure it out from the file.open(..) modes

Advice gratefully received.

* UPDATE

My method to write P2P pieces ended up as follows (for those who may glean some value from it)

def writePiece(self, filename, pieceindex, bytes, ipsrc, ipdst, ts): 
    file = open(filename,"r+b")
    if not self.piecemap[ipdst].has_key(pieceindex):
        little = struct.pack('<'+'B'*len(bytes), *bytes) 
        # Seek to offset based on piece index 
        file.seek(pieceindex * self.piecesize)
        file.write(little)
        file.flush()
        self.procLog.info("Wrote (%d) bytes of piece (%d) to %s" % (len(bytes), pieceindex, filename))

    # Remember we have this piece now in case duplicates arrive 
    self.piecemap[ipdst][pieceindex] = True
    file.close()

Note: I also addressed some endian issues using struct.pack which plagued me for a while.

For anyone wondering, the project I am working on is to analyse BT messages captured directly off the wire.

Answer 1

You opened the file in append ("a") mode. All writes are going to the end of the file, irrespective of the calls to seek().

Answer 2

Try using 'r+b' rather than 'ab'.

Answer 3

>>> import os
>>> filename = 'tempfile'
>>> def write(filename,data,offset):
...     try:
...         f = open(filename,'r+b')
...     except IOError:
...         f = open(filename,'wb')
...     f.seek(offset)
...     f.write(data)
...     f.close()
...
>>> write(filename,'1' * (1024*32),1024*1024)
>>> write(filename,'1' * (1024*32),0)
>>> os.path.getsize(filename)
1081344

Answer 4

It seems to me like there's not a lot of point in trying to assemble the file until all the pieces of it are there. Why not keep the pieces separate until all are present, then write them to the final file in order? That's what most P2P apps do, AFAIK.