Profiling python C extensions

Question

I have developed a python C-extension that receives data from python and compute some cpu intensive calculations. It's possible to profile the C-extension?

The problem here is that writing a sample test in C to be profiled would be challenging because the code rely on particular inputs and data structures (generated by python control code).

Do you have any suggestions?

Answer 1

With gprof, you can profile any program that was properly compiled and linked (gcc -pg etc, in gprof's case). If you're using a Python version not built with gcc (e.g., the Windows precompiled version the PSF distributes), you'll need to research what equivalent tools exist for that platform and toolchain (in the Windows PSF case, maybe mingw can help). There may be "irrelevant" data there (internal C functions in the Python runtime), and, if so, the percentages shown by gprof may not be applicable -- but the absolute numbers (of calls, and durations thereof) are still valid, and you can post-process gprof's output (e.g., with a little Python script;-) to exclude the irrelevant data and compute the percentages you want.

Answer 2

I've found my way using google-perftools. The trick was to wrap the functions StartProfiler and StopProfiler in python (throught cython in my case).

To profile the C extension is sufficient to wrap the python code inside the StartProfiler and StopProfiler calls.

from google_perftools_wrapped import StartProfiler, StopProfiler
impor c_extension # extension to profile c_extension.so

StartProfiler("output.prof")
... calling the interesting functions from the C extension module ...
StopProfiler()

Then to analyze for example you can export in callgrind format and see the result in kcachegrind:

pprof --callgrind c_extension.so output.prof > output.callgrind 
kcachegrind output.callgrind