Python extensions - performance

views:

answers:

Python extensions - performance

Hi all,

I am using Boost.Python to extend python program functionality. Python scripts do a lot of calls to native modules so I am really concerned about the performance of python-to-cpp type conversion and data marshaling.

I decided to try exposing methods natively through Python C API. May be somebody already tried that before ? Any success ... at least in theory ?

The problem I run into is that how to convert PyObject* back to class instance, PyArg_parse provides O& option, but what I am looking is simply a pointer to C++ object in memory... how can I get it in function ?

if ( PyArg_ParseTuple(args, "O", &pyTestClass ) ) { // how to get TestClass from pyTestClass ?? }

Thanks

+1 A:

I haven't tried Boost.Python, but I've extended Python using raw C as well as Cython. I recommend Cython; if you're careful enough you can get code with the same efficiency as raw C but with a lot less boilerplate code.

Regarding efficiency, it's relative. It depends on what you want to do and how you do it. For example, what I've done very often is write the inner loop of some image processing or matrix operation in C, and have this function be called by Python with pointers to matrices as arguments. The matrices themselves don't get copied, so the overhead is minimal.

dimatura 2010-08-02 04:01:12

Lets bring some abstraction. I have a simple function like that: def sum( a, b ): return a + b. Let's now imagine it's called from C for hundreds of thousands of times so each microsecond worth here. Obviously this is just an abstraction, i.e. I know it doesn't make sense to do so. In real life I have a better example though. So what I need here is to get the minimum delay between calls, means I need to optimize the time between calls of the real function sum that does what it should. What is the best approach in that situation ?

Alex 2010-08-02 22:33:06

So you want to call Python from C? Well anyway, the overhead in the interface between Python and C comes mainly from having to box/unbox things into objects (e.g. in C you have an int, in Python you have an integer which is much bigger) and handling the reference counting. It's 'fast', but if the function to be called is very simple and must be called in a tight loop there might be too much overhead. To get an idea of the work involved, Cython has a option to write an html file that shows the C code generated for Cython/Python line. That gives you a sense of the cost of each operation.

dimatura 2010-08-04 05:50:32

ansaurus

tags:

views:

answers:

Python extensions - performance

related questions