slow File Processing in python

Question

Hello All,

I am trying a file operation using python.Aim is to continuously read a file of size(100bytes),pack and send them through socket. These files are read from a directory.

Problem: when i run the program continuously, execution time is increasing. Initially, execution time is less than a second; later it reaches till 8~10seconds. I am not able get exact reason for the delay.If anyone can throw some light on the issue, it will be more helpful.

Here i have attached my code...

def handlefile(filename):
        for sat in range(len(Numfiles)):
                filename = 
                fsize = os.path.getsize(filename)
                if fsize != 100:
                        continue
                rfile = open(filename,'rb')
                text = rfile.read()
                msg = struct.unpack("<100b",text)
                for i in range(len(msg)):
                        packMessage  = packMessage + struct.pack("<b",msg[i])
                print "time:",datetime.datetime.now() - startTime

The file are binary files.

Initial time taken : 671 ms

on executing continuously for more than 10 times,time increases slowly. Last few values, 671ms . . . . 9.879 ms 88.686 ms 135.954 ms

I am using python-2.5.4 version.

If anyone had come across similar problem. Please provide me some inputs.

Thanks Das

Answer 1

Have you checked the number of file-handles your process has open? You may want yo use the with-statement to make sure they get closed when not needed anymore:

with open(filename, 'rb') as rfile:
    text = rfile.read()
    # etc.

When the with-block is left, the file will closed automatically.

Answer 2

From what I see, packMessage is growing monotonically:

packMessage  = packMessage + struct.pack("<b",msg[i])

If you repeat it many times, it may grow big, consume a lot of memory, and at some point your application may become much slower. Try looking at top or htop when you run your program (in top, press M to sort by memory allocation, f to add resident memory field).

Also opening and reading the same file every time is not the best solution from the performance point of view. Consider reading it only once before entering the loop.

Answer 3

Hi,

I have left rfile.close() while posting. Here i have attached the code with output,

def Handlefile(packMessage):
        msg =[]
        startTime = time.time()
        #pdb.set_trace()                                
        for sat in range(len(Numfiles)):
                start = datetime.datetime.now()
                fname = 
                filename = ".".join([fname , "ubx"])
                try :
                        fsize = os.path.getsize(filename)
                except os.error:
                        print "WAR:File Doesnot Exist"
                if fsize != 100:
                        continue
                rfile = open(filename,'rb')
                #with open(filename,'rb') as rfile:
                text = rfile.read()
                packMessage  = packMessage + text
                rfile.close()     
                print "Time diff:",datetime.datetime.now() - start
        endTime = time.time()
        print "Running took %.3f seconds"%(endTime - startTime) 
        return packMessage

This function is called in a while loop. so it will be run continuously.I have added timer for testing.

On execution,the output is

1st Time,

Time diff: 0:00:00.075588
Time diff: 0:00:00.317736
Time diff: 0:00:00.014908
Time diff: 0:00:00.041021
Time diff: 0:00:00.011927
Time diff: 0:00:00.252999
Time diff: 0:00:00.090901
Time diff: 0:00:00.256198
Time diff: 0:00:00.015685
Running took 1.077 seconds

5th time,

Time diff: 0:00:00.318296
Time diff: 0:00:00.259274
Time diff: 0:00:00.236204
Time diff: 0:00:00.237291
Time diff: 0:00:00.317070
Time diff: 0:00:00.575538
Time diff: 0:00:00.308405
Time diff: 0:00:00.313486
Time diff: 0:00:00.136546
Running took 2.902 seconds

10th time,

Time diff: 0:00:01.306515
Time diff: 0:00:00.046437
Time diff: 0:00:00.471991
Time diff: 0:00:00.658932
Time diff: 0:00:00.460947
Time diff: 0:00:00.268992
Time diff: 0:00:00.169916
Time diff: 0:00:00.115676
Time diff: 0:00:00.076303
Running took 3.576 seconds

As per the output, First time the function takes 1sec and it keeps increasing every time the function is called.

The file i am reading is updated every 1sec.Hence i need to read the file everytime while processing.Thus, i need to do this operation every 1sec.

My operation is success only 1st time and rest of time it fails.

Please post your suggestion on this problem.

Thanks for the reply