Python C-API Object Initialisation

Question

What is the correct way to initialise a python object into already existing memory (like the inplace new in c++)

I tried this code however it causes an access violation with a debug build because the _ob_prev and _ob_next are not set..

//PyVarObject *mem; -previously allocated memory

Py_INCREF(type);
//couldnt get PyObject_HEAD_INIT or PyVarObject_HEAD_INIT to compile
//however the macros resolve to this
PyVarObject init = {{_PyObject_EXTRA_INIT 1, ((_typeobject*)type)}, 0};
*mem = init;
//...other init code for type...

The crash occures on line 1519 in object.c

void
_Py_ForgetReference(register PyObject *op)
{
#ifdef SLOW_UNREF_CHECK
        register PyObject *p;
#endif
    if (op->ob_refcnt < 0)
        Py_FatalError("UNREF negative refcnt");
    if (op == &refchain ||
        op->_ob_prev->_ob_next != op || op->_ob_next->_ob_prev != op) { //----HERE----//
        fprintf(stderr, "* ob\n");
        _PyObject_Dump(op);
        fprintf(stderr, "* op->_ob_prev->_ob_next\n");
        _PyObject_Dump(op->_ob_prev->_ob_next);
        fprintf(stderr, "* op->_ob_next->_ob_prev\n");
        _PyObject_Dump(op->_ob_next->_ob_prev);
        Py_FatalError("UNREF invalid object");
    }
#ifdef SLOW_UNREF_CHECK
    for (p = refchain._ob_next; p != &refchain; p = p->_ob_next) {
        if (p == op)
            break;
    }
    if (p == &refchain) /* Not found */
        Py_FatalError("UNREF unknown object");
#endif
    op->_ob_next->_ob_prev = op->_ob_prev;
    op->_ob_prev->_ob_next = op->_ob_next;
    op->_ob_next = op->_ob_prev = NULL;
    _Py_INC_TPFREES(op);
}

Answer 1

Taking your question at face value, you have a few options. The quick and dirty method is to put an extra Py_INCREF into your initialisation code. Assuming you have no refcount bugs, the refcount will never return to zero, deallocation code will never be called, and there should be no crash. (In fact this may be the way that the statically allocated builtin type objects are managed!)

You could write a custom allocator and deallocator for your type that manages memory in your preferred manner. You could in fact write a custom allocator and deallocator for the whole python interpreter.

You could manage your python objects in the normal way but store pointers in them to data in the memory you are managing yourself.

Looking at the bigger picture... why are you trying to do this?

Also, your comments that

I tried this code however it causes an access violation with a debug build because the _ob_prev and _ob_next are not set..

and

//couldnt get PyObject_HEAD_INIT or PyVarObject_HEAD_INIT to compile
//however the macros resolve to this

are worrying! Have you successfully defined a type which uses standard memory management, before moving on to more advanced stuff?

Answer 2

You can look at Py_NoneStruct for how this is done. Your code looks basically right.

It is a refcounting bug. Statically allocated objects can never be freed, so _Py_ForgetReference should never be called.

If you want to be able to free them you must use use a custom allocator instead of static initialization.

Answer 3

What are doing is pretty horrible. Unless this code path is really performance critical I'd advise you to allocate your objects on the heap as is normally done.