What are the best (powerful yet easy) languages to implement text/data processing and code generation?

Question

If I want to write my engine which will generate all the code solving the task described in simple declarative style, what languages should I look at?

Answer 1

Prolog. Definitely Prolog. I know it's not the vanilla option, so here is the rationale:

• Prolog has a flexible syntax that can be made even more flexible by using its interpret-time macro expansion mechanism (a.k.a term expansion).
• In case that the native syntax won't do, Prolog has a good built-in parsing mechanism: Definite Clauses Grammar (DCG).
• Prolog is intended for finding solutions based on declarations.
• Prolog has several useful libraries for declarative computing, such as constraint solving, linear equations solver, etc.
• Prolog searches for all solutions, thus making variations and optimizations more natural.
• Prolog uses flexible data structures (functors) which it can both examine and generate, so generating complex structures in rather natural. You don't need string generation: You can generate functors and then print them. DCG also helps with this.

I actually did this sort of projects: Generators from natural-looking Prolog to languages like SQL and Erlang. Getting to know Prolog takes some time, but in my experience it's worth your while.

Answer 2

That's an extremely broad topic, so it deserves an extremely broad answer.

An engine designed to implement arbitrary processing and generation of code is the DMS Software Reengineering Toolkit. DMS parses a wide variety of langauges, will accept definitons of more languages (including specification or modelling languages), provides pattern matching and transformation using declarative patterns written at source level syntax, etc.

DMS isn't a single language; rather, it is a set of Domain-Specific Languages (DSLs) each of which provides support for one of the issues that a code metaprogramming tool must address: langauge grammars, attribute computations, pattern matching/transformation, flow analysis, task scripting.

DMS is extremely powerful; it has been used to build many code analysis, generation, and transformation tools (you can see a wide variety of examples at the web site offered as COTS tools). It is not necessarily easy, because analyzing/transforming code for real langauges such as Java, C# and C++ is complicated, because those languages are complicated, and because the fundamental problem of transforming code from one level of abstraction to another and producing optimized results is fundamentally complicated. I'll claim DMS is as easy as practical for the problem targeted.

(Full disclosure: I'm the principal behind DMS).