Combining C and Python functions in a module

Question

I have a C extension module, to which I would like to add some Python utility functions. Is there a recommended way of doing this?

For example:

import my_module

my_module.super_fast_written_in_C()
my_module.written_in_Python__easy_to_maintain()

I'm primarily interested in Python 2.x.

Answer 1

Prefix your native extension with an underscore. Then, in Python, create a wrapper module that imports that native extension and adds some other non-native routines on top of that.

Answer 2

The usual way of doing this is: mymod.py contains the utility functions written in Python, and imports the goodies in the _mymod module which is written in C and is imported from _mymod.so or _mymod.pyd. For example, look at .../Lib/csv.py in your Python distribution.

Answer 3

The existing answers describe the method most often used: it has the potential advantage of allowing pure-Python (or other-language) implementations on platforms in which the compiled C extension is not available (including Jython and IronPython).

In a few cases, however, it may not be worth splitting the module into a C layer and a Python layer just to provide a few extras that are more sensibly written in Python than in C. For example, gmpy (lines 7113 ff at this time), in order to enable pickling of instances of gmpy's type, uses:

copy_reg_module = PyImport_ImportModule("copy_reg");
if (copy_reg_module) {
    char* enable_pickle =
        "def mpz_reducer(an_mpz): return (gmpy.mpz, (an_mpz.binary(), 256))\n"
        "def mpq_reducer(an_mpq): return (gmpy.mpq, (an_mpq.binary(), 256))\n"
        "def mpf_reducer(an_mpf): return (gmpy.mpf, (an_mpf.binary(), 0, 256))\n"
        "copy_reg.pickle(type(gmpy.mpz(0)), mpz_reducer)\n"
        "copy_reg.pickle(type(gmpy.mpq(0)), mpq_reducer)\n"
        "copy_reg.pickle(type(gmpy.mpf(0)), mpf_reducer)\n"
    ;
    PyObject* namespace = PyDict_New();
    PyObject* result = NULL;
    if (options.debug)
        fprintf(stderr, "gmpy_module imported copy_reg OK\n");
    PyDict_SetItemString(namespace, "copy_reg", copy_reg_module);
    PyDict_SetItemString(namespace, "gmpy", gmpy_module);
    PyDict_SetItemString(namespace, "type", (PyObject*)&PyType_Type);
    result = PyRun_String(enable_pickle, Py_file_input,
                          namespace, namespace);

If you want those few extra functions to "stick around" in your module (not necessary in this example case), you would of course use your module object as built by Py_InitModule3 (or whatever other method) and its PyModule_GetDict rather than a transient dictionary as the namespace in which to PyRun_String. And of course there are more sophisticated approaches than to PyRun_String the def and class statements you need, but, for simple enough cases, this simple approach may in fact be sufficient.