Is it reasonable to integrate python with c for performance?

Question

Hi, I like to use python for almost everything and always had clear in my mind that if for some reason I was to find a bottleneck in my python code(due to python's limitations), I could always use a C script integrated to my code.

But, as I started to read a guide on how to integrate python. In the article the author says:

There are several reasons why one might wish to extend Python in C or C++, such as:

• Calling functions in an existing library.
• Adding a new builtin type to Python
• Optimising inner loops in code
• Exposing a C++ class library to Python
• Embedding Python inside a C/C++ application

Nothing about performance. So I ask again, is it reasonable to integrate python with c for performance?

Answer 1

* Optimising inner loops in code

Isn't that about performance ?

Answer 2

Performance is a broad topic so you should be more specific. If the bottleneck in your program involves a lot of networking then rewriting it in C/C++ probably won't make a difference since it's the network calls taking up time, not your code. You would be better off rewriting the slow section of your program to use fewer network calls thus reducing the time your program spends waiting on entwork IO. If your doing math intensive stuff such as solving differential equations and you know there are C librarys that can offer better performance then the way you are currently doing it in Python you may want to rewrite the section of your program to use those librarys to increase it's performance.

Answer 3

In my experience it is rarely necessary to optimize using C. I prefer to identify bottlenecks and improve algorithms in those areas completely in Python. Using hash tables, caching, and generally re-organizing your data structures to suit future needs has amazing potential for speeding up your program. As your program develops you'll get a better sense of what kind of material can be precalculated, so don't be afraid to go back and redo your storage and algorithms. Additionally, look for chances to kill "two birds with one stone", such as sorting objects as you render them instead of doing huge sorts.

When everything is worked to the best of your knowledge, I'd consider using an optimizer like Psyco. I've experienced literally 10x performance improvements just by using Psyco and adding one line to my program.

If all else fails, use C in the proper places and you'll get what you want.

Answer 4

You will gain a large performance boost using C from Python (assuming your code is well written, etc) because Python is interpreted at run time, whereas C is compiled beforehand. This will speed up things quite a bit because with C, your code is simply running, whereas with Python, the Python interpreter must figure out what you are doing and interpret it into machine instructions.

Answer 5

C can definitely speed up processor bound tasks. Integrating is even easier now, with the ctypes library, or you could go for any of the other methods you mention.

I feel mercurial has done a good job with the integration if you want to look at their code as an example. The compute intensive tasks are in C, and everything else is python.

Answer 6

I've been told for the calculating portion use C for the scripting use python. So yes you can integrate both. C is capable of faster calculations than that of python

Answer 7

The C extensions API is notoriously hard to work with, but there are a number of other ways to integrate C code.

For some more usable alternatives see http://www.scipy.org/PerformancePython, in particular the section about using Weave for easy inlining of C code.

Also of interest is Cython, which provides a nice system for integrating with C code. Cython is used for optimization by some well-respected high-performance Python projects such as NumPy and Sage.

As mentioned above, Psyco is another attractive option for optimization, and one which requires nothing more than

import psyco
psyco.bind(myfunction)

Psyco will identify your inner loops and automatically substitute optimized versions of the routines.