tags:

views:

68

answers:

1

Firstly, I am working on a windows xp 64 machine with 4gb ram and 2.29 ghz x4

I am indexing 220,000 lines of text that are more or less the same length. These are divided into 15 equally sized files. File 1/15 takes 1 minute to index. As the script indexes more files, it seems to take much longer with file 15/15 taking 40 minutes.

My understanding is that the more I put in memory, the faster the script is. The dictionary is indexed in a hash, so fetch operations should be O(1). I am not sure where the script would be hanging the CPU.

I have the script here.

+1  A: 

You can try to monitor your machine to see if you're running out of memory. If so, you may want to look for memory leaks in your code.

pocketfullofcheese
I agree. If your machine runs out of memory and you're hitting the swap, chances are your script is going to do things like take 40 minutes per file.
mattbasta