Lexical analysis and Macros

Question

I'm writing a demo compiler for a toy programming language in C.

What problems can arise if we do macro processing in a separate phase between reading the program and lexical analysis?

Answer 1

"Preporcessor" generaly means a tool that transforms the codes before the main compiler gets a hold of it.

That is the preprocessor lexes (and possibly parses) the source by its rules, performs some transformation and outputs a result. The main compiler lexes and parses the result of the preprocessor's work according to its rules (which may be different from those used by the preprocessor).

So if the preprocessor works non-trivial changes on the program text it can be hard to predict the final result at a glance. This is true of c with the c-preporcessor, but (or prehaps thus) convention holds that you should only use the preprocessor in a few ways that have fairly predictable results. (I believe that the c preprocessor is Turing complete, so there is no limit to what can be achieved if you don't go insane trying to debug it. On thinking about Pavel's comment, I concede no loops, no recursion, no stacks. Seems like that kills it. Thanks. You can still go bonkers trying to debug a sufficiently advance pile of macros.)

Answer 2

As mentioned above, the C preprocessor is quite limited since it only performs basic text transformations, due to restrictions in it's language. On the other hand, looking at the macro system in Common Lisp, you can see the advantage of having a macro system integrated into the main language, since it lets you use the primary language facilities in the macro.

A simple example

(defmacro ntimes (data n)
  `(loop for i from 1 to ,n collecting ,data))

(print (ntimes 'a 10))

Result : (A A A A A A A A A A)

This transformation will be done at compile time (one of the nice things of the source being it's own AST). This something which could not be done by a seperate preprocessor (unless the preprocessor contained a copy of the compiler!)