Resuming a large file write in Python

Question

I have a big file transfer (say 4gb or so) and rather than using shutil, I'm just opening and writing it the normal file way so I can include a progress percentage as it moves along.

It then occurred to me to try to attempt to resume the file write, if for some reason it borked out during the process. I haven't had any luck though. I presumed it would be some clever combination of offsetting the read of the source file and using seek, but I haven't had any luck so far. Any ideas?

Additionally, is there some sort of dynamic way to figure what block size to use when reading and writing files? I'm fairly novice to that area, and just read to use a larger size for larger file (I'm using 65536 at the moment). Is there a smart way to do it, or does one simply guess..? Thanks guys.

Here is the code snippet of the appending file transfer:

                newsrc = open(src, 'rb')
                dest_size = os.stat(destFile).st_size
                print 'Dest file exists, resuming at block %s' % dest_size
                newsrc.seek(dest_size)
                newdest = open(destFile, 'a')
                cur_block_pos = dest_size
                # Start copying file
                while True:
                    cur_block = newsrc.read(131072)                    
                    cur_block_pos += 131072
                    if not cur_block:
                        break
                    else:
                       newdest.write(cur_block)

It does append and start writing, but it then writes dest_size more data at the end than it should for probably obvious reasons to the rest of you. Any ideas?

Answer 1

For the second part of your question, data is typically read from and written to a hard drive in blocks of 512 bytes. So using a block size that is a multiple of that should give the most efficient transfer. Other than that, it doesn't matter much. Just keep in mind that whatever block size you specify is the amount of data that the I/O operation stores in memory at any given time, so don't choose something so large that it uses up a lot of your RAM. I think 8K (8192) is a common choice, but 64K should be fine. (I don't think the size of the file being transferred matters much when you're choosing the best block size)