What language should we use to let people extend our terminal/sniffer program?

Question

Hello,

We have a very versatile terminal/sniffer application which can do all sorts of things with TCP, UDP and serial connections.

We are looking to make it extensible -- i.e, allow people to write their own protocol parsers, highlighters, etc.

We created a C-like language for extending the product, and then discovered that for some coders, this presents a steep learning curve.

We are now pondering the question: Should we stick to C or go with something like Ruby or Lua?

C is beautiful for low-level stuff (like parsing binary data), because it supports pointers. But for exactly that reason, it can be tough to learn.

Ruby (etc) are easy to learn, but don't have pointers, so anything that has to do with parsing binary data gets ugly very fast.

What do you think? For extending a product that parses binary data -- Ruby/Lua or C/C++?

Would be great if you could give some background when you respond -- especially if you've done something similar.

Answer 1

If you have an API written does it make a difference? The person using the C-like API would only have to understand the difference between passing by value or reference.

Answer 2

Your core does one thing very good, so fine. Let it be that way. I think you should create an API based on std in/out, just like the way of good unix design. Then anyone can extend it in any language of choice.

Answer 3

Tcl was designed with the goal to allow scripting for C programs, so it would be much easier to implement.

http://en.wikipedia.org/wiki/Tcl#Interfacing_with_other_languages

Answer 4

I second Johan's idea. Although in past when I had to do something like this I stuck to C language APIs and people were restricted to use C language only. But now I look at it, I realize that it would have been more efficient if we would have done the way Johan describes

PS: And by coincidence it was a protocol testing app using packet sniffer

Answer 5

perl, sed, awk, lex, antler, ... These are languages I'm somewhat familiar with that I'd like to write something like this in. It depends on the data flow, though.

It's really hard to say what the correct thing to use is. Something that I don't think anyone else has mentioned is to keep in mind that the scripts will have bugs. It's very easy to design something like this in such a way that bugs in the scripts (especially run time errors) either just show up a "error in script" or kill the whole system.

You should keep that the scripts should be unit testable and that failures should be reproducible.

Answer 6

Wireshark, the "world's foremost network protocol analyzer", is also a packet sniffer/analyzer, formerly also called Ethereal. It uses Lua to enable writing custom dissectors and taps, see the manual.

However, note that I have not used it, so I cannot tell how nice/effective/easy to learn the API is.

Answer 7

Like TCL, Lua was designed to be tightly integrated with an application. Personally, I find Lua's syntax and idioms to be much easier to deal with than TCL.

Lua is easy to integrate with an existing system, and easy to extend. It is also fairly easy to create safe sandboxes in which user-supplied code can run without full access to the innards of your product.

Answer 8

I don't think it matters what you do as long as you do one thing, drop the in-house language. It sounds like you choose to make C into a scripting language. One issue I see with this is it will look familiar to C programmers, but not be the same. I can't imagine you have mimicked the semantics of C that would make existing C programmers comfortable. And as you have mentioned, others will find it hard to learn.

The company I am working at have developed their own language. It uses XML for structure so parsing is easy. The language grows "as needed." Meaning if a feature is missing then it will be added. I'm pretty sure it went from an XML database to something that needed control flow. But my point is that if you aren't thinking about building it as a language, then you'll be limiting what users can do with it unintentionally.

Personally I've been looking at how I can get the company to start taking advantage of Lua. And specifically Lua for several reasons. Lua was developed as an extension language that was general purpose. It easily interfaces with the language, including Python and Ruby. It is small and simple for use by non-programmers (not really needed in your case). It is simple enough to replace XML, INI... for configuration settings and powerful enough to replace the need for another programming language.

http://www.lua.org/spe.html