Selecting An Embedded Language

Question

I'm making an application that analyses one or more series of data using several different algorithms (agents). I came to the idea that each of these agents could be implemented as separate Python scripts which I run using either the Python C API or Boost.Python in my app.

I'm a little worried about runtime overhead TBH, as I'm doing some pretty heavy duty data processing and I don't want to have to wait several minutes for each simulation. I will typically be making hundreds of thousands, if not millions, of iterations in which I invoke the external "agents"; am I better of just hardcoding everything in the app, or will the performance drop be tolerable?

Also, are there any other interpreted languages I can use other than Python?

Answer 1

you could probably create an embedded language using C++ templates and operator overloading, see for example ublas or ftensor matrix languages. i do not think python or other interpreted languages of is suitable for having numbercrunching/data processing.

Answer 2

For millions of calls (from I'm assuming c++, because you mentioned boost) into python, yes: you will notice a performance hit. This may or may not be significant - perhaps the speed gain of trying out new 'agents' would be greater than the hit. Python does have fast numerical libraries (such as numpy) that might help, but you'll still incur overhead of marshalling data, calling into python, the gil, etc.

Yes, you can embed many other languages: check out lua. Also, check out swig.org, which can connect to many other languages besides python.

Answer 3

Yes, tons. Lua and Python seems to be the most popular:

Embedding Lua

• http://www.lua.org/pil/24.html
• http://stackoverflow.com/questions/38338/why-is-lua-considered-a-game-language
• http://stackoverflow.com/questions/250151/lua-as-a-general-purpose-scripting-language
• http://stackoverflow.com/questions/250151/lua-as-a-general-purpose-scripting-language

Embedding Python

• http://docs.python.org/extending/embedding.html

Embedding Ruby

• http://stackoverflow.com/questions/239315/how-to-embed-ruby-in-c

Embed Perl

• http://perldoc.perl.org/perlembed.html

Embed JavaScript

• http://spiderape.sourceforge.net/

There are dozens of JavaScript engines around, this is just an example. Some of them are also frighteningly quick.

Answer 4

I will typically be making hundreds of thousands, if not millions, of iterations in which I invoke the external "agents"

The performance drop will be noticeable, perhaps painful. If you can put the data into arrays and process it in batches using NumPy, it should be much faster.

NumPy makes it super easy to do any kind of arithmetic a million times in a row. For example, squaring every element of an array is like this:

>>> x = numpy.array([1, 2, 3, 4, 5, 6, 7])
>>> x**2
array([1, 4, 9, 16, 25, 36, 49])

Super easy, and the tight inner loop here is actually implemented in C.

Of course NumPy can also do more advanced number-crunching.

Answer 5

Tcl was designed from the ground up to be an embedded language.

Answer 6

I believe tcl and Rexx were both intended for this purpose.