Tracking file load progress in Python

Question

A lot of modules I use import entire files into memory or trickle a file's contents in while they process it. I'm wondering if there's any way to track this sort of loading progress? Possibly a wrapper class that takes a callback?

Answer 1

I would go by this by determining the size of the file, and then simply dividing the total by the number of bytes read. Like this:

import os

def show_progress(file_name, chunk_size=1024):
    fh = open(file_name, "r")
    total_size = os.path.getsize(file_name)
    total_read = 0
    while True:
        chunk = fh.read(chunk_size)
        if not chunk: 
            fh.close()
            break
        total_read += len(chunk)
        print "Progress: %s percent" % (total_read/total_size)
        yield chunk

for chunk in show_progress("my_file.txt"):
    # Process the chunk
    pass

Edit: I know it isn't the best code, but I just wanted to show the concept.

Answer 2

If you actually mean "import" (not "read") then you can override the import module definitions. You can add timing capabilities.

See the imp module.

If you mean "read", then you can trivially wrap Python files with your own file-like wrapper. Files don't expose too many methods. You can override the interesting ones to get timing data.

>>> class MyFile(file):
...     def read(self,*args,**kw):
...         # start timing
...         result= super(MyFile,self).read(*args,**kw)
...         # finish timing
...         return result