Linking languages

Question

Hi,

I asked a question earlier about which language to use for an AI prototype. The consensus seemed to be that if I want it to be fast, I need to use a language like Java or C++, but that Python / Perl / Ruby would be good for the interface bits.

So, this leads me on to another question. How easy is it to link these languages together? And which combination works best? So, if I wanted to have a Ruby CGI-type program calling C++ or Java AI functions, is that easy to do? Any pointers for where I look for information on doing that kind of thing? Or would a different combination be better?

My main experience with writing web applications started with C++ CGI and then moved on to Java servlets (about 10 years ago) and then after a long gap away from programming I did some PHP. But I've not had experience of writing a web application in a scripting language which then calls out to a compiled language for the speed-critical bits. So any advice will be welcome!

Thanks,

Ben

Answer 1

It may be a good approach to start with a script, and call a compilation-based language from that script only for more advanced needs.

For instance, calling java from ruby script works quite well.

require "java"
# The next line exposes Java's String as JString
include_class("java.lang.String") { |pkg, name| "J" + name }
s = JString.new("f")

Answer 2

You can build your program in one of the higher level languages for example Python or Ruby and then call modules that are compiled in the lower level language for the parts you need performance. You can choose a platform depending on the lower level language you want.

For example if you want to do C++ for the speedy stuff you can just use plain Python or Ruby and call DLLs compiled in C++. If you want to use Java you can use Jython or one of the other dynamic languages on the Java platform to call the Java code this is easier than the C++ route because you've got a common virtual machine so a Java object can be used directly in Jython or JRuby. The same can be done on the .Net platform with the Iron-languages and C# although you seem to have more experience with C++ and Java so those would be better options.

Answer 3

First, a meta comment: I would highly recommend coding the entire thing in a high-level language, profiling like mad, and optimizing only where profiling shows it's necessary. First optimize the algorithm, then the code, then think about bringing in the heavy iron. Having an optimum algorithm and clean code will make things much easier when/if you need to reimplement in a lower-level language.

Speaking for Python, IronPython/C# is probably the easiest optimization path.

CPython with C++ is doable, but I find C a lot easier to handle (but not all that easy, being C). Two tools that ease this are cython/pyrex (for C) and shedskin (for C++). These compile Python into C/C++, and from there you can access C/C++ libraries without too much ado.

I've never used jython, but I hear that the jython/Java optimization path isn't all that bad.

Answer 4

Boost.Python provides an easy way to turn C++ code into Python modules. It's rather mature and works well in my experience.

For example, the inevitable Hello World...

char const* greet()
{
  return "hello, world";
}

can be exposed to Python by writing a Boost.Python wrapper:

#include <boost/python.hpp>

BOOST_PYTHON_MODULE(hello_ext)
{
  using namespace boost::python;
  def("greet", greet);
}

That's it. We're done. We can now build this as a shared library. The resulting DLL is now visible to Python. Here's a sample Python session:

>>> import hello_ext
>>> print hello.greet()
hello, world

(example taken from boost.org)

Answer 5

I agree with the Idea of coding first in a high level language such as Python, Profiling and then Implementing any code that needs speeding up in C / C++ and wrapping it for use in the high level language.

As an alternative to boost I would like to suggest SWIG for creating Python callable code from C. Its reasonably painless to use, and will compile callable modules for a wide range of languages. (Python, Ruby, Java, Lua. to name a few) from C code.

The wrapping process is semi automated, so there is no need to add new functions to the base C code, making a smoother work flow.

Answer 6

If you choose Perl there are plenty of resources for interfacing other languages.

Inline::C
Inline::CPP
Inline::Java

From Inline::C-Cookbook:

use Inline C => <<'END_C';

  void greet() {
    printf("Hello, world\n");
  }
END_C

greet;

Answer 7

I have a different perspective, having had lots of luck with integrating C++ and Python for some real time live video image processing.

I would say you should match the language to the task for each module. If you're responding to a network, do it in Python, Python can keep up with network traffic just fine. UI: Python, People are slow, and Python is great for UIs using wxPython or PyObjC on Mac, or PyGTK. If you're doing math on lots of data, or signal processing, or image processing... code it in C or C++ with unit tests, then use SWIG to create the binding to any higher level language.

I used the image libraries in wxWidgets in my C++, which are already exposed to Python through wxPython, so it was extremely powerful and quick. SCONS is a build tool (like make) which knows what to do with swig's .i files.

The topmost level can be in C or Python, you'll have more control and fewer packaging and deployment issues if the top level is in C or C++... but it will take a really long time to duplicate what Py2EXE or Py2App gives you on Windows or Mac (or freeze on Linux.)

Enjoy the power of hybrid programming! (I call using multiple languages in a tightly coupled way 'hybrid' but it's just a quirk of mine.)

Answer 8

If the problem domain is hard (and AI problems can often be hard), then I'd choose a language which is expressive or suited to the domain first, and then worry about speeding it up second. For example, Ruby has meta-programming primitives (ability to easily examine and modify the running program) which can make it very easy/interesting to implement certain types of algorithms.

If you implement it in that way and then later need to speed it up, then you can use benchmarking/profiling to locate the bottleneck and either link to a compiled language for that, or optimise the algorithm. In my experience, the biggest performance gain is from tweaking the algorithm, not from using a different implementation language.