how to speed up the code??

views:

143

answers:

+1 Q:

how to speed up the code??

in my program i have a method which requires about 4 files to be open each time it is called,as i require to take some data.all this data from the file i have been storing in list for manupalation. I approximatily need to call this method about 10,000 times.which is making my program very slow?

any method for handling this files in a better ways and is storing the whole data in list time consuming what is better alternatives for list?

I can give some code,but my previous question was closed as that only confused everyone as it is a part of big program and need to be explained completely to understand,so i am not giving any code,please suggest ways thinking this as a general question...

thanks in advance

+2 A:

Opening, closing, and reading a file 10,000 times is always going to be slow. Can you open the file once, do 10,000 operations on the list, then close the file once?

Pace 2010-06-09 14:32:09

no way,i each time i open a different file..

kaushik 2010-06-09 15:35:02

+1 A:

It might be better to load your data into a database and put some indexes on the database. Then it will be very fast to make simple queries against your data. You don't need a lot of work to set up a database. You can create an SQLite database without requiring a separate process and it doesn't have a complicated installation process.

Mark Byers 2010-06-09 14:32:41

+3 A:

As a general strategy, it's best to keep this data in an in-memory cache if it's static, and relatively small. Then, the 10k calls will read an in-memory cache rather than a file. Much faster.

If you are modifying the data, the alternative might be a database like SQLite, or embedded MS SQL Server (and there are others, too!).

It's not clear what kind of data this is. Is it simple config/properties data? Sometimes you can find libraries to handle the loading/manipulation/storage of this data, and it usually has it's own internal in-memory cache, all you need to do is call one or two functions.

Without more information about the files (how big are they?) and the data (how is it formatted and structured?), it's hard to say more.

FrustratedWithFormsDesigner 2010-06-09 14:33:20

I have about 3200 files each of around 400kb..for each call 4 of these files are to be used..I will just use the file data will not manupalate data in file.i just need to read them.it is just properties data each line having 5 words..

kaushik 2010-06-09 15:33:20

i find your idea interesting, how to store then in cache?

kaushik 2010-06-09 15:34:05

Well, I am guessing your data looks something like this:`property 1 = word1 word2 word3 word4 word5`. You could create a hash-table where `property1` is the key to the value `word1 word2 word3 word4 word5`. Setup the hash table when the app starts up, then every other time you need the data, access the hash-table instead of the files.

FrustratedWithFormsDesigner 2010-06-09 15:42:17

...Of course, if the structure has more meaning than just `property key=value`, you might want a more complex data structure. For example, instead of just storing the words as a single string, you might want to store them as an array of words. It will depend on your specific needs, but I hope that gives the general idea.

FrustratedWithFormsDesigner 2010-06-09 15:43:33

do i need to drop data from all my 3200 files into hash table..for this to i guess need to open 3200 files.

kaushik 2010-06-09 15:44:35

Yes, but you'd only open each one once, at the begining of the program, INSTEAD of for each of your 10,000 calls that need the data. Alternatively, you could have a lazy-load cache, where you only open a file and load the data if you discover the data you need is not in the cache.

FrustratedWithFormsDesigner 2010-06-09 15:55:28

Call the open to the file from the calling method of the one you want to run. Pass the data as parameters to the method

Derek 2010-06-09 14:40:32

If the files are structured, kinda configuration files, it might be good to use ConfigParser library, else if you have other structural format then I think it would be better to store all this data in JSON or XML and perform any necessary operations on your data

sultan 2010-06-09 14:44:53

ansaurus

tags:

views:

answers:

how to speed up the code??

related questions