Dynamic and Lexical variables in Common Lisp

Question

I am reading the book 'Practical Common Lisp' by Peter Seibel.

In Chapter 6, "Variables" sections "Lexical Variables and Closures" and "Dynamic, a.k.a. Special, Variables". http://www.gigamonkeys.com/book/variables.html

My problem is that the examples in both sections show how (let ...) can shadow global variables and doesn't really tell the difference between the Dynamic and Lexical vars.

I understand how closures work but I don't really get whats so special about let in this example:

(defvar *x* 10)

(defun foo ()
  (format t "Before assignment~18tX: ~d~%" *x*)
  (setf *x* (+ 1 *x*))
  (format t "After assignment~18tX: ~d~%" *x*))


(defun bar ()
  (foo)
  (let ((*x* 20)) (foo))
  (foo))


CL-USER> (foo)
Before assignment X: 10
After assignment  X: 11
NIL


CL-USER> (bar)
Before assignment X: 11
After assignment  X: 12
Before assignment X: 20
After assignment  X: 21
Before assignment X: 12
After assignment  X: 13
NIL

I feel like there is nothing special is going on here. The outer foo in bar increments the global x, and foo surrounded by let in bar increments the shadowed x. What's the big deal? I don't see how is this supposed to explain the difference between lexical and dynamic vars. Yet the book continues like this:

So how does this work? How does LET know that when it binds x it's supposed to create a dynamic binding rather than a normal lexical binding? It knows because the name has been declared special.12 The name of every variable defined with DEFVAR and DEFPARAMETER is automatically declared globally special.

What whould happen if let would bind x using "normal lexical binding"? All in all, what are the differences between dynamic and lexical binding and how is this example special regarding dynamic binding?

Answer 1

When a variable is lexically scoped, the system looks to where the function is defined to find the value for a free variable. When a variable is dynamically scoped, the system looks to where the function is called to find the value for the free variable. Variables in Common Lisp are all lexical by default; however, dynamically scoped variables can be defined at the top level using defvar or defparameter.

A simpler example

lexical scoping (with setq):

(setq x 3)

(defun foo () x)

(let ((x 4)) (foo)) ; returns 3

dynamic scoping (with defvar):

(defvar x 3)

(defun foo () x)

(let ((x 4)) (foo)) ; returns 4

How does the let know if a variable is lexical or dynamic? It doesn't. On the other hand, when foo goes to find the value of X, it will initially find the lexical value defined at the top level. It then checks to see if the variable is supposed to be dynamic. If it is, then foo looks to the calling environment, which, in this case, uses let to overshadow the value of X to be 4.

(note: this is an oversimplification, but it will help to visualize the difference between the different scoping rules)

Answer 2

Maybe this example will help.

;; the lexical version

(let ((x 10)) 
  (defun lex-foo ()
    (format t "Before assignment~18tX: ~d~%" x)
    (setf x (+ 1 x))
    (format t "After assignment~18tX: ~d~%" x)))

(defun lex-bar ()
  (lex-foo)
  (let ((x 20)) ;; does not do anything
    (lex-foo))
  (lex-foo))

;; CL-USER> (lex-bar)
;; Before assignment X: 10
;; After assignment  X: 11
;; Before assignment X: 11
;; After assignment  X: 12
;; Before assignment X: 12
;; After assignment  X: 13

;; the dynamic version

(defvar *x* 10)
(defun dyn-foo ()
  (format t "Before assignment~18tX: ~d~%" *x*)
  (setf *x* (+ 1 *x*))
  (format t "After assignment~18tX: ~d~%" *x*))

(defun dyn-bar()
  (dyn-foo)
  (let ((*x* 20))
    (dyn-foo))
  (dyn-foo))

;; CL-USER> (dyn-bar)
;; Before assignment X: 10
;; After assignment  X: 11
;; Before assignment X: 20
;; After assignment  X: 21
;; Before assignment X: 11
;; After assignment  X: 12

;; the special version

(defun special-foo ()
  (declare (special *y*))
  (format t "Before assignment~18tX: ~d~%" *y*)
  (setf *y* (+ 1 *y*))
  (format t "After assignment~18tX: ~d~%" *y*))

(defun special-bar ()
  (let ((*y* 10))
    (declare (special *y*))
    (special-foo)
    (let ((*y* 20))
      (declare (special *y*))
      (special-foo))
    (special-foo)))

;; CL-USER> (special-bar)
;; Before assignment X: 10
;; After assignment  X: 11
;; Before assignment X: 20
;; After assignment  X: 21
;; Before assignment X: 11
;; After assignment  X: 12

Answer 3

You can tell your Lisp to bind local variables dynamically, too:

(let ((dyn 5))
  (declare (special dyn))
  ... ;; DYN has dynamic scope for the duration of the body
  )

Answer 4

You say: feel like there is nothing special is going on here. The outer foo in bar increments the global x, and foo surrounded by let in bar increments the shadowed x. What's the big deal?

The 'special' that's going on here is that LET can shadow the value of *x*. With lexical variables that's not possible.

The code declares *x* to be special via the DEFVAR.

In FOO now the value of *x* is looked up dynamic. FOO will take the current dynamic binding of *x* or, if there is none, the symbol value of the symbol *x*. A new dynamic binding can, for example, be introduced with LET.

A lexical variable on the other hand has to be present in the lexical environment somewhere. LET, LAMBDA, DEFUN and others can introduce such lexical variables:

(let ((x 3))
  (* (sin x) (cos x)))

(lambda (x)
  (* (sin x) (cos x)))

(defun baz (x)
  (* (sin x) (cos x)))

If our code were:

(defvar x 0)

(let ((x 3))
  (* (sin x) (cos x)))

(lambda (x)
  (* (sin x) (cos x)))

(defun baz (x)
  (* (sin x) (cos x)))

Then X were special in all cases, because of the DEFVAR declaration, which declares X to be special. Because of this, there is the convention to declare special variables as *X*. Thus only variables with stars around them are special - by convention. That's a useful convention.

In your code you have then:

(defun bar ()
  (foo)
  (let ((*x* 20))
    (foo))
  (foo))

Since *x* has be declared special via the DEFVAR, the LET construct introduces a new dynamic binding for *x*. FOO is then called. Since inside FOO the *x* uses dynamic binding, it looks up the current one and finds that *x* is dynamically bound to 20.

The value of a special variable is found in the current dynamic binding.

(defun foo-s ()
  (declare (special *x*))
  (+ *x* 1))

If the variable has been declared special by a DEFVAR or DEFPARAMETER, then the local special declaration can be omitted.

A lexical variable directly references the variable binding.

(defun foo-l (x)
  (+ x 1))

Let's see it in practice:

(let ((f (let ((x 10))
           (lambda ()
             (setq x (+ x 1))))))
  (print (funcall f))    ; form 1
  (let ((x 20))          ; form 2
    (print (funcall f))))

Here variables are lexical. In form 2 the LET will not shadow the X in our function f. It can't. The function uses the lexical bound variable, introduced by the LET ((X 10) . Surrounding the call with another lexical bound X in form 2 has no effect on our function.

Let's try special variables:

(let ((f (let ((x 10))
           (declare (special x))
           (lambda ()
             (setq x (+ x 1))))))
  (print (funcall f))    ; form 1
  (let ((x 20))          ; form 2
    (declare (special x))
    (print (funcall f))))

What now? Does that work?

It does not!

The first form calls the function and it tries to look up the dynamic value of X and there is none.

We get an error in form 1 that X is unbound, because there is no dynamic binding in effect.

Form 2 would work, since the LET with the special declaration introduces a dynamic binding for X.