Create a DSL vs. Embed an Existing Language

Question

This often comes up: your application has gotten extensive enough that it's time to add some programmability into it to make it flexible. One example might be a finance application - you want to add a formula editor so that you can create your own custom formulas without having to re-compile the code.

You have to make a choice: do you create your own tokenizer, parser, and interpreter/compiler chain, something which might take a long time and might be done incorrectly? Or do you just embed another scripting language, which has the problem that it'll likely bloat your code and expose your app to security vulnerabilities.

How would you balance the trade-offs and make this decision?

Answer 1

There are no trades offs -- embed a throughly tested, well documented interpretor. Otherwise, you'll end up with an abomination like MAXScript.

Answer 2

I would create my own interpreter using an existing parser generator (like ANTLR or Haskell/Scala's parser combinators). It's really not as hard as all that, and for simple languages, it's really quite easy. I create the implementation for a devilishly simple DSL in an afternoon and it ran perfectly the first time (no bugs).

With that said, you don't want to have to design your own Turing Complete language. If your needs are that sophisticated, you should probably embed a scripting language. If you're on the JVM, JRuby and Clojure are excellent candidates for this sort of thing, particularly given their own strengths in the area of internal DSLs.

Answer 3

How about a plug-in system? There are a few advantages:

• Allow customer developers to develop in the environment in which the source app is developed.
• In modern dev. platforms, you gain a lot of control and security through software contracts.
• If it's designed correctly, you can properly blame bloat and failure to the plug-in that caused it - like Chrome does when Flash or another third party plug-in crashes.
• Easy to add extra security through licensing/certs.
• Easy to merge a plug-in with the app if it's amazing and you want all of your customers to have it.

Answer 4

Unless the DSL is simple enough that the parser/interpreter fits on a single page, I would recommend embedding an existing scripting language.

I recently spent several months working on a project I inherited that contained a completely home grown scripting language. I invested a lot of time in understanding the parser & interpreter, so that I could fix bugs, make it thread safe, extend it, optimize it. Plus there was the time to learn and understand the quirks of this new scripting language that woked almost-like-but-not-quite-the-same-as others I already knew. I would much rather have used that time to embed an existing language like Ruby or Lua, and tuning it to fit our needs.

The user would have benefited from a language that was easier to program in, with less quirks and gotcha's. I would have benefited from a deeper understanding of the internals of a well-designed & popular language, instead of gaining relativly worthless expert knowledge in 'myScript'.