how to improve the performance??

Question

i had prepared a project on making a software application. It is complete and working fine except that the speed of execution is very slow..i have taken several chunks of code and optimized it..

i tried psyco..ie i installed psyco and added two lines on the top of my code import psyco psyco.full()

Don't know whether this is the way using psyco..if this is wrong.please tell me how to use psyco..coz i added this and found no improvement..

i have tried profiling and i know the code lines taking time but these cant be further optimized and are unavoidable line of code..

i also thought of option of rewriting the code in 'c' using some python package..but i always had a very bad experience in using additional package of python which are not part of basic python..

i am using python 2.6 and windows vista..please kindly tell methods method for increasing the speed of execution of the whole code significantly..atleast 5x times..please..

i havent writen my code in method,there few method in between thou..there is no main..

yes as few suggested my is an IO bound problem..as i need to call the code some 500 times and this involves opening and closing of files of atleast 2 per call.. Ans here when opening a .pm file, it has two collumms and i need the first collums only ,so i am coping the entire first columns into the list and passing it to a function to get its row number and then opening other file to get the elements of that row number into a list... this is the task i wanted...i guess loading the elements of first columns into the list is time consumming any idea to rectify this.. how can i improve the performance for IO bound bottlenecks looking for help desperately

Answer 1

psyco can be used as simple as import and call psyco.full(). so you are right about your psyco usage.

If you are trying to build a python module using C/C++, have a look at boost::python

You should really post your code for further analysis.

Answer 2

You are unlikely to see a 5x performance difference by just tweaking the code around.

First you should look at improving your algorithm - are you using the best datastructures for the job? Perhaps using a dict or a set in the right place can speed your code up a alot.

Writing a C module is not all that hard, and is another option if you can find no way to improve the Python code. Usually you would expect more than a 5x speed up by using C code.

Maybe your problem is IO bound. Then you need to look at ways to improve the performance of the IO

If you want more help here, you'll probably have to show some code or at least describe what your program does.

UPDATE: Looks like you are opening and closing lots of files which tends to be painfully slow on windows.

Answer 3

To optimize your code for speed you simply have to profile it and see where the problem is. Guessing does not help. But once you know where, the most bang for your buck usually come from those in descending order: improving algorithm, using more appropriate data structures, removing resource bottlenecks (io,memory,cpu), reducing memory allocation, reducing context switching (processes and subroutines).

Answer 4

Here's one opportunity for optimization: you're calling get_list twice, with very similar arguments:

join_cost_index_end[index] = get_list(file, float(abs1), fout)
join_cost_index_strt[index] = get_list(file, float(abs2), fout)

That means that most of the work in get_list is being done twice for no good reason. Rewrite it so that get_list is being called once, and have it return both index_end and index_strt at the same time.

Answer 5

You could get a lot better performance if you could switch to binary file formats. Most of your code is doing parsing and string manipulation. You're doing a lot of converting strings to floats, which is slower than you think.

Answer 6

why bot just try using cython? You should get much better performance without changing any of the code. With a little bit of modification this should help even more.