What are good references about testing a domain specific language?

Question

I am creating a new domain specific language. How do you test a language, or something you can do (transform, compile, check) with a language. What are good references or starting points for learning about best practices?

Answer 1

This might be completely unrelated to what you were trying to ask, but PHP comes with automated test scripts, so that anyone can test that basic language functionality (variables, 'if' statements, loops, etc) are working correctly.

Answer 2

The traditional way is to bootstrap your implementation in the language you're implementing. That way you get a large test suite for "free" (relatively, of course), and you have a strong incentive to fix problems and optimize.

Answer 3

You can build programming languages the test-first way:

1. Create a script in the language you are creating
2. Assert that the script works by investigating its side effects. You can do this by e.g. creating unit tests in the host language (the machine language) that execute the script in the new language.

If you want to learn more about how to create programming languages, "The Definitive ANTLR Reference" book is a good starting point.

A very helpful pattern to test the syntax of a new language is the following:

1. Interpret a script and build an AST
2. Generate code in the new language from the AST
3. Generate an AST from the generated code
4. Assert that both AST structures are identical

Sometimes it's more convenient to assert that generated code from two ASTs are identical.

Build your tests like you build your language:

1. lexical
2. syntactic
3. semantic

Answer 4

I like to hijack working unit tests from other projects. For example, if I know the boost unit tests are working, then I can just re-run the test after XYZ new compile-time operations and make sure they still pass. I'm not sure what's available for other languages, but IMO for C++, boost is the definitive compiler stress test.

For a new language, I'd subject it to my standard evaluation procedure:

• Solve the Project Euler problems in the new language.
• Build a unit test suite that runs down the solutions you wrote for those problems and make sure the compiled code still returns the correct answers.

It's far from perfect, but it will force you to build a wide range of algorithms/strategies, and the result is a compact validation test. If you make a change in the language specification, it wouldn't be hard to update your solutions to accomodate them.

Answer 5

The first thing to research is unit testing because this is something that developers should do right from the beginning. In fact some people even advocate writing unit tests before you write any actual code.

There are lots of unit testing frameworks around for most popular languages like Java, Python and so on. This blog post gives a comprehensive list of the unit testing frameworks for Javascript.

Next step is to learn about lint programs which examine a codebase for incorrect or unclear constructions. This is often not included in a build process, but is a valuable part of testing. There are lint programs for C, Javascript and Python among others.