What's the most efficient way to share large amounts of data between Python and C++

Question

I'm writing a system that allows python scripts to be executed within a C++ application.

The python scripts are used to modify values within arrays of data (typically 2048x2048x4 arrays of floats)

I'm currently using numpy arrays created using the array API and registered with Python.

In the python scripts I'm accessing the arrays like

for all x in range(0,Width):
    for all y in range(0,Height)
        Array[x][y][0] = blah
        Array[x][y][1] = blah
        Array[x][y][2] = blah
        Array[x][y][3] = blah

This seems to be pretty slow. Is there a quicker way to set this up?

Answer 1

You should probably see if you can set several values in the array in one step rather than in four (or more) steps.

I believe the ellipsis syntax may be useful here:

http://stackoverflow.com/questions/118370/how-do-you-use-the-ellipsis-slicing-syntax-in-python

Or else something like:

Array[x,y,:]=blah

or

Array[x,y,:]=blah1,blah2,blah3,blah4

if you have different values for each element.

That's assuming that the bottle-neck in your code is due to the number of assignments you are doing. It's probably worth doing some profiling to see exactly what's being slow. Try the same code, but without actually storing the results to see if it's much faster. If not then it's probably not the assignment that's slow...

Answer 2

You might want to have a look at Boost.Python. It focuses on making C++ code available in Python, but it also provides exec and eval functions that should allow you to efficiently interact with python code.

Answer 3

I thought I would suggest numpy, but you're already using it. I'm afraid that leaves domain-specific changes to do less work. As John Montgomery mentions, you'll need to figure out what is taking the time in the Python code, and determine if you can avoid some of it.

Are there patterns to the work being done in Python? Perhaps you can provide domain-specific helper functions (written in C) that can be called from the Python code.