How far can LISP macros go?

Question

I have read a lot that LISP can redefine syntax on the fly, presumably with macros. I am curious how far does this actually go? Can you redefine the language structure so much that it borderline becomes a compiler for another language? For example, could you change the functional nature of LISP into a more object oriented syntax and semantics, maybe say having syntax closer to something like Ruby?

Especially, is it possible to get rid of the parenthesis hell using macros? I have learned enough (Emacs-)LISP to customize Emacs with my own micro-features, but I am very curious how far macros can go in customizing the language.

Answer 1

If you want lisp to look like Ruby use Ruby.

It's possible to use Ruby (and Python) in a very lisp like way which is one of the main reasons they have gained acceptance so quickly.

Answer 2

@sparkes

Sometimes LISP is the clear language choice, namely Emacs extensions. I'm sure I could use Ruby to extend Emacs if I wanted to, but Emacs was designed to be extended with LISP, so it seems to make sense to use it in that situation.

Answer 3

@mike but if you are comfortable in emacs then parenthesis hell is your penance :)

/me ducks from the people who can't leave a comment with personality at zero.

Answer 4

It's a tricky question. Since lisp is already structurally so close to a parse tree the difference between a large number of macros and implementing your own mini-language in a parser generator isn't very clear. But, except for the opening and closing paren, you could very easily end up with something that looks nothing like lisp.

Answer 5

Regular macros operate on lists of objects. Most commonly, these objects are other lists (thus forming trees) and symbols, but they can be other objects such as strings, hashtables, user-defined objects, etc. These structures are called s-exps.

So, when you load a source file, your Lisp compiler will parse the text and produce s-exps. Macros operate on these. This works great and it's a marvellous way to extend the language within the spirit of s-exps.

Additionally, the aforementioned parsing process can be extended through "reader macros" that let you customize the way your compiler turns text into s-exps. I suggest, however, that you embrace Lisp's syntax instead of bending it into something else.

You sound a bit confused when you mention Lisp's "functional nature" and Ruby's "object-oriented syntax". I'm not sure what "object-oriented syntax" is supposed to be, but Lisp is a multi-paradigm language and it supports object-oriented programming extremelly well.

BTW, when I say Lisp, I mean Common Lisp.

I suggest you put your prejudices away and give Lisp an honest go.

Answer 6

I'm not a Lisp expert, heck I'm not even a Lisp programmer, but after a bit of experimenting with the language I came to the conclusion that after a while the parenthesis start becoming 'invisible' and you start seeing the code as you want it to be. You start paying more attention to the syntactical constructs you create via s-exprs and macros, and less to the lexical form of the text of lists and parenthesis.

This is specially true if you take advantage of a good editor that helps with the indentation and syntax coloring (try setting the parenthesis to a color very similar to the background).

You might not be able to replace the language completely and get 'Ruby' syntax, but you don't need it. Thanks to the language flexibility you could end having a dialect that feels like you are following the 'Ruby style of programming' if you want, whatever that would mean to you.

I know this is just an empirical observation, but I think I had one of those Lisp enlightenment moments when I realized this.

Answer 7

Yes, you can fundamentally change the syntax, and even escape "the parentheses hell". For that you will need to define a new reader syntax. Look into reader macros.

I do suspect however that to reach the level of Lisp expertise to program such macros you will need to immerse yourself in the language to such an extent that you will no longer consider parenthese "hell". I.e. by the time you know how to avoid them, you will have come to accept them as a good thing.

Answer 8

That's a really good question.

I think it's nuanced but definitely answerable:

Macros are not stuck in s-expressions. See the LOOP macro for a very complex language written using keywords (symbols). So, while you may start and end the loop with parentheses, inside it has its own syntax.

Example:

(loop for x from 0 below 100
      when (even x)
      collect x)

That being said, most simple macros just use s-expressions. And you'd be "stuck" using them.

But s-expressions, like Sergio has answered, start to feel right. The syntax gets out of the way and you start coding in the syntax tree.

As for reader macros, yes, you could conceivably write something like this:

#R{
      ruby.code.goes.here
  }

But you'd need to write your own Ruby syntax parser.

You can also mimic some of the Ruby constructs, like blocks, with macros that compile to the existing Lisp constructs.

#B(some lisp (code goes here))

would translate to

(lambda () (some lisp (code goes here)))

See this page for how to do it.

Answer 9

Parenthesis hell? I see no more parenthesis in:

(function toto)

than in:

function(toto);

And in

(if tata (toto)
  (titi)
  (tutu))

no more than in:

if (tata)
  toto();
else
{
  titi();
  tutu();
}

I see less brackets and ';' though.

Answer 10

Yes, you can redefine the syntax so that Lisp becomes a compiler. You do this using "Reader Macros," which are different from the normal "Compiler Macros" that you're probably thinking of.

Common Lisp has the built-in facility to define new syntax for the reader and reader macros to process that syntax. This processing is done at read-time (which comes before compile or eval time). To learn more about defining reader macros in Common Lisp, see the Common Lisp Hyperspec -- you'll want to read Ch. 2, "Syntax" and Ch. 23, "Reader". (I believe Scheme has the same facility, but I'm not as familiar with it -- see the Scheme sources for the Arc programming language).

As a simple example, let's suppose you want Lisp to use curly braces rather than parentheses. This requires something like the following reader definitions:

;; { and } become list delimiters, along with ( and ).
(set-syntax-from-char #\{ #\( )
(defun lcurly-brace-reader (stream inchar) ; this was way too easy to do.
  (declare (ignore inchar))
  (read-delimited-list #\} stream t))
(set-macro-character #\{ #'lcurly-brace-reader)

(set-macro-character #\} (get-macro-character #\) ))
(set-syntax-from-char #\} #\) )

;; un-lisp -- make parens meaningless
(set-syntax-from-char #\) #\] ) ; ( and ) become normal braces
(set-syntax-from-char #\( #\[ )

You're telling Lisp that the { is like a ( and that the } is like a ). Then you create a function (lcurly-brace-reader) that the reader will call whenever it sees a {, and you use set-macro-character to assign that function to the {. Then you tell Lisp that ( and ) are like [ and ] (that is, not meaningful syntax).

Other things you could do include, for example, creating a new string syntax or using [ and ] to enclose in-fix notation and process it into S-expressions.

You can also go far beyond this, redefining the entire syntax with your own macro characters that will trigger actions in the reader, so the sky really is the limit. This is just one of the reasons why Paul Graham and others keep saying that Lisp is a good language in which to write a compiler.

Answer 11

Over and over again, newcomers to Lisp want to "get rid of all the parenthesis." It lasts for a few weeks. No project to build a serious general purpose programming syntax on top of the usual S-expression parser ever gets anywhere, because programmers invariably wind up preferring what you currently perceive as "parenthesis hell." It takes a little getting used to, but not much! Once you do get used to it, and you can really appreciate the plasticity of the default syntax, going back to languages where there's only one way to express any particular programming construct is really grating.

That being said, Lisp is an excellent substrate for building Domain Specific Languages. Just as good as, if not better than, XML.

Good luck!

Answer 12

see this example of how reader macros can extend the lisp reader with complex tasks like XML templating:

http://common-lisp.net/project/cl-quasi-quote/present-class.html

this user library compiles the static parts of the XML into UTF-8 encoded literal byte arrays at compile time that are ready to be write-sequence'd into the network stream. and they are usable in normal lisp macros, they are orthogonal... the placement of the comma character influences which parts are constant and which should be evaluated at runtime.

more details available at: http://common-lisp.net/project/cl-quasi-quote/

another project that for Common Lisp syntax extensions: http://common-lisp.net/project/cl-syntax-sugar/

Answer 13

What you are asking is somewhat like asking how to become an expert chocolatier so that you can remove all that hellish brown stuff from your favourite chocolate cake.

Answer 14

Scheme macros are very powerful. I wrote a complete LINQ implementation as Scheme macro a while back.

https://ironscheme.svn.codeplex.com/svn/IronScheme/IronSchemeConsole/ironscheme/linq.ss

With all this power, I am running out of good ideas to apply it!

Answer 15

@S.Harrison (sorry can't add a comment)

This is a bit nit-picky, but you have be be careful with terminology or risk confusing people searching the documentation. "Compiler macro" means something specific in (common) lisp, and it is not a `normal' macro! You have macros, compiler macros, and reader macros to keep straight, but the first is the most often used.

Answer 16

One of the uses of macros that blew my mind was the compile-time verification of SQL requests against DB.

Once you realize you have the full language at hand at compile-time, it opens up interesting new perspectives. Which also means you can shoot yourself in the foot in interesting new ways (like rendering compilation not reproducible, which can very easily turn into a debugging nightmare).

Answer 17

The best explanation of Lisp macros I have ever seen is at

http://video.google.com/videoplay?docid=448441135356213813

starting at about 55 minutes in. This is a video of a talk given by Peter Seibel, the author of "Practical Common Lisp", which is the best Lisp textbook there is.

The motivation for Lisp macros is usually hard to explain, because they really come into their own in situations that are too lengthy to present in a simple tutorial. Peter comes up with a great example; you can grasp it completely, and it makes good, proper use of Lisp macros.

You asked: "could you change the functional nature of LISP into a more object oriented syntax and semantics". The answer is yes. In fact, Lisp originally didn't have any object-oriented programming at all, not surprising since Lisp has been around since way before object-oriented programming! But when we first learned about OOP in 1978, we were able to add it to Lisp easily, using, among other things, macros. Eventually the Common Lisp Object System (CLOS) was developed, a very powerful object-oriented programming system that fits elegantly into Lisp. The whole thing can be loaded as an extension -- nothing is built-in! It's all done with macros.

Lisp has an entirely different feature, called "reader macros", that can be used to extend the surface syntax of the language. Using reader macros, you can make sublanguages that have C-like or Ruby-like syntax. They transform the text into Lisp, internally. These are not used widely by most real Lisp programmers, mainly because it is hard to extend the interactive development environment to understand the new syntax. For example, Emacs indentation commands would be confused by a new syntax. If you're energetic, though, Emacs is extensible too, and you could teach it about your new lexical syntax.

Answer 18

Peter Norvig made a Prolog Interpreter in his book.