C++ and Embedded Python - NUL Terminated Strings

Question

I'm working on embedding Python 2.6 into an existing c++ application. So far I have the Libraries linked in and am able to successfully initialize the Python Interpreter and can also transfer data to Python. I'm having trouble retrieving it, and hope someone can steer me the right direction. I'm working with this:

Py_Initialize();

pModule = PyImport_ImportModule("cBuffers"); // This crashes after 1st call.
pDict = PyModule_GetDict(pModule);
pClass = PyDict_GetItemString(pDict, "rf_pdf");
pMeth = PyString_FromString("main");

if (PyCallable_Check(pClass) && PyClass_Check(pClass)) {
  pInstance = PyInstance_New(pClass, NULL, NULL);
  pOutput = PyObject_CallMethodObjArgs(pInstance, pMeth, pOpts, pInput, NULL);
}

if (pOutput != NULL) {
  string pPdf = PyString_AsString(pOutput);
  Py_DECREF(pOutput);
} else {
  PyErr_Print();
}

// Cleanup
Py_DECREF(pModule);
Py_DECREF(pModule); // Has an extra reference, not positive why.
Py_DECREF(pMeth);
Py_DECREF(pInstance);
Py_DECREF(pOpts);
Py_DECREF(pInput);

Py_Finalize();

pOpts and pInput are both generated using PyString_FromString earlier in the code. The trouble I'm having is that when I attempt to retrieve the output using PyString_AsString the return value is NUL Terminated. Unfortunately, because I'm generating PDF Documents, NULs are not only allowed, they're almost guaranteed. Can anyone tell me how I return String Data from Python back to C++ without ending at the first NUL it encounters?

As an additional question, This code can be called multiple times as a part of a background service that's creating PDF Documents from incoming Print Data. The first time this code is called into it works as expected. Any subsequent calls fail at the indicated line just after Py_Initialize(). Help on how to determine what's going on there would be most appreciated as well. Thanks in advance,

Answer 1

A few points:

• Don't use strings. You might even be able to make them work here with some contortions on *_StringAndSize() functions, but it won't be what you want. You should store your data in a custom data structure (or a buffer) that is just a sequence of bytes (do you really want clients performing string operations on this data in Python?). If your object really is a buffer object, you should use the Buffer API.

• Your imported module has a refcount of 2 because it's being held in sys.modules (for efficiency for the next time you try to import it). Never decref references you don't own or you'll crash your program. The Importing Modules section of the documentation should really cover this, but it doesn't.

• It's pretty expensive to initialize Python and tear it down every time you do these operations. You should try to reorganize your use case such that you can call Py_Initialize only once when your application starts (or the first time it needs Python), and then only call Py_Finalize when your application is definitely done with Python, or when it quits.

• You're being very lazy with error checking - most of the Python C/API functions can return NULL to indicate that an exception has been thrown, and you're almost never checking this value. If something fails you're going to start crashing in very odd places. You can read about this in the Exception Handling section of the C/API manual.