In python, under what circumstances is SWIG a better choice than ctypes for calling entry points in shared libraries? Let's assume you don't already have the SWIG interface file(s).
What are the performance metrics of the two?
views:
3619answers:
6
+7
A:
CTypes is very cool and much easier than SWIG, but it has the drawback that poorly or malevolently-written python code can actually crash the python process. You should also consider boost python. IMHO it's actually easier than swig while giving you more control over the final python interface. If you are using C++ anyway, you also don't add any other languages to your mix.
David Nehme
2008-09-25 20:35:17
Oooooh! Shiny new thing I didn't know about -- thanks for the pointer to Boost.Python!!
Kevin Little
2008-09-25 20:45:40
Kevin, I'm actually surprised more people haven't heard of boost::python. I was at a talk on CTypes in New York Python user's group and they hadn't heard of it either. I guess a lot of python folks try to avoid C++ as much as possible.
David Nehme
2008-09-25 20:50:50
+14
A:
SWIG generates (rather ugly) C or C++ code. It is straightforward to use for simple functions (things that can be translated directly) and reasonably easy to use for more complex functions (such as functions with output parameters that need an extra translation step to represent in Python.) For more powerful interfacing you often need to write bits of C as part of the interface file. For anything but simple use you will need to know about CPython and how it represents objects -- not hard, but something to keep in mind.
ctypes allows you to directly access C functions, structures and other data, and load arbitrary shared libraries. You do not need to write any C for this, but you do need to understand how C works. It is, you could argue, the flip side of SWIG: it doesn't generate code and it doesn't require a compiler at runtime, but for anything but simple use it does require that you understand how things like C datatypes, casting, memory management and alignment work. You also need to manually or automatically translate C structs, unions and arrays into the equivalent ctypes datastructure, including the right memory layout.
It is likely that in pure execution, SWIG is faster than ctypes -- because the management around the actual work is done in C at compiletime rather than in Python at runtime. However, unless you interface a lot of different C functions but each only a few times, it's unlikely the overhead will be really noticeable.
In development time, ctypes has a much lower startup cost: you don't have to learn about interface files, you don't have to generate .c files and compile them, you don't have to check out and silence warnings. You can just jump in and start using a single C function with minimal effort, then expand it to more. And you get to test and try things out directly in the Python interpreter. Wrapping lots of code is somewhat tedious, although there are attempts to make that simpler (like ctypes-configure.)
SWIG, on the other hand, can be used to generate wrappers for multiple languages (barring language-specific details that need filling in, like the custom C code I mentioned above.) When wrapping lots and lots of code that SWIG can handle with little help, the code generation can also be a lot simpler to set up than the ctypes equivalents.
Thomas Wouters
2008-09-25 20:47:28
Was struggling with SWIG and came across this answer. It convinced me to switch to CTypes. [Why didn't I think to look on stackoverflow first ;-)]A good overview of may be found at: http://www.slideshare.net/gnunify/c-types-extending-python
CyberED
2009-08-25 02:20:31
I much prefer CTypes because it avoids compilation altogether. This is particularly advantageous when writing a module that might be used on multiple platforms, and particularly those that don't have easy access to a compiler (such as 64-bit Windows). CTypes is also python-version-agnostic. That is, you can write a ctypes interface, and it can work under Python 2.4, 2.6, and 3.1 without modification.
Jason R. Coombs
2009-09-10 16:14:12
+5
A:
You can also use Pyrex, which can act as glue between high-level Python code and low-level C code. lxml is written in Pyrex, for instance.
Torsten Marek
2008-09-25 20:54:22
"Cython is a language that makes writing C extensions for the Python language as easy as Python itself. Cython is based on the well-known Pyrex, but supports more cutting edge functionality and optimizations."
mtasic
2009-08-11 17:23:17
+5
A:
ctypes is great, but does not handle C++ classes. I've also found ctypes is about 10% slower than a direct C binding, but that will highly depend on what you are calling.
If you are going to go with ctypes, definitely check out the Pyglet and Pyopengl projects, that have massive examples of ctype bindings.
Peter Shinners
2008-09-25 21:22:15
+1
A:
I'm going to be contrarian and suggest that, if you can, you should write your extension library using the standard Python API. It's really well-integrated from both a C and Python perspective... if you have any experience with the Perl API, you will find it a very pleasant surprise.
Ctypes is nice too, but as others have said, it doesn't do C++.
How big is the library you're trying to wrap? How quickly does the codebase change? Any other maintenance issues? These will all probably affect the choice of the best way to write the Python bindings.
Dan
2008-09-26 04:49:05
@Dan, the libraries I'm dealing with are third-party -- VMware's VIX API, for example. I have no choice but to use them as best I can. I use the standard Python API whenever possible, believe me! :)
Kevin Little
2008-09-26 14:41:28
+3
A:
In my experience, ctypes does have a big disadvantage: when something goes wrong (and it invariably will for any complex interfaces), it's a hell to debug.
The problem is that a big part of your stack is obscured by ctypes/ffi magic and there is no easy way to determine how did you get to a particular point and why parameter values are what they are..