Wrapping C++ dynamic array with Python+ctypes, segfault

Question

Hi!

I wanted to wrap a small C++ code allocating an array with ctypes and there is something wrong with storing the address in a c_void_p object.

(Note: the pointers are intentionally cast to void*, 'cause later I want to do the allocation the same way for arrays of C++ objects, too.)

The C(++) functions to be wrapped:

void* test_alloc()
{
    const int size = 100000000;
    int* ptr = new int[size];
    std::cout << "Allocated " << size * sizeof(int) << " bytes @ " <<
                 ptr << std::endl;
    return static_cast<void*>(ptr);
}

void test_dealloc(void* ptr)
{
    int* iptr = static_cast<int*>(ptr);
    std::cout << "Trying to free array @ " << iptr << std::endl;
    delete[] iptr;
}

The Python wrapper (assume the former functions are already imported with ctypes):

class TestAlloc(object):
    def __init__(self):
        self.pointer = ctypes.c_void_p(test_alloc())
        print "self.pointer points to ", hex(self.pointer.value)

    def __del__(self):
        test_dealloc(self.pointer)

For small arrays (e.g. size = 10), it seems ok:

In [5]: t = TestAlloc()
Allocated 40 bytes @ 0x1f20ef0
self.pointer points to  0x1f20ef0

In [6]: del t
Trying to free array @ 0x1f20ef0

But if I want to allocate a large one (size = 100 000 000), problems occur:

In [2]: t = TestAlloc()
Allocated 400000000 bytes @ 0x7faec3b71010 
self.pointer points to  0xffffffffc3b71010L

In [3]: del t
Trying to free array @ 0xffffffffc3b71010
Segmentation fault

The address stored in ctypes.c_void_p is obviously wrong, the upper 4 bytes are invalid. Somehow 32-bit and 64-bit addresses are mixed, and with the large array allocation the memory manager (in this case) is forced to return an address not representable on 32 bits (thx TonJ).

Can someone please provide a workaround for this?

The code has been compiled with g++ 4.4.3 and run on Ubuntu Linux 10.04 x86_64 with 4G RAM. Python version is 2.6.5.

Thank you very much!

UPDATE:

I managed to solve the problem. I forgot to specify restype for test_alloc(). The default value for restype was ctypes.c_int, into which the 64-bit address did not fit. By also adding a test_alloc.restype = ctypes.c_void_p before the call of test_alloc() solved the problem.

Answer 1

From just looking at it, it seems that the problem is not in the small/big array allocation, but in a mix of 32bit and 64bit addresses. In your example, the address of the small array fits in 32 bits, but the address of the big array doesn't.