Lexical and dynamic scoping in Mathematica: Local variables with Module, With, and Block

Question

The following code returns 14 as you'd expect:

Block[{expr},
  expr = 2 z;
  f[z_] = expr;
  f[7]]

But if you change that Block to a Module then it returns 2*z. It seems to not matter what other variables besides expr you localize. I thought I understood Module, Block, and With in Mathematica but I can't explain the difference in behavior between Module and Block in this example.

Related resources:

• Tutorial on Modularity and the Naming of Things from the Mathematica documentation
• Excerpt from a book by Paul R. Wellin, Richard J. Gaylord, and Samuel N. Kamin
• Explanation from Dave Withoff on the Mathematica newsgroup

PS: Thanks to Michael Pilat, Davorak, and Bill White for following the scent-trail on this weirdness. Davorak clarifies and gets to the heart of the issue here: http://stackoverflow.com/questions/2739643/why-would-mathematica-break-normal-scoping-rules-in-module

Answer 1

Have you used Trace on both expressions?

Answer 2

First off I think you have exposed a bug here.

Second I think I can offer some insight in to why this is happening, keeping in mind my knowledge of the internals of mathematica are limited.

A statement like: f[z_] := 2 z in Full form is:

SetDelayed[f[Pattern[z, Blank[]]], 2 z]

This sets the DownValue[f] to:

{HoldPattern[f[z_]] :> 2 z}

Then later when an expression, like f[2], is later is evaluated something like following is being preformed:

f[2] /. HoldPattern[f[z_]] :> 2 z

Which would evaluate to 4. Now this is all possible because pattern matching is happening with Pattern[z, Blank[]] from the first code block. This works even if you have perviously set z to a number. In other words.

z = 5;
f[z_] := 2*z

Still produces the same downvalues for f:

{HoldPattern[f[z_]] :> 2 z}

This is possible because Pattern has the HoldFirst Attribute.

The HoldFirst Attribute is not enough protection if you evaluate this inside a Module. Example:

SetAttributes[tmp, HoldFirst];
Module[{expr},
 expr = 2 z;
 tmp[expr]
]

outputs:

tmp[expr$8129]

I propose that because HoldFirst Attribute does not provide immunity to Module's variable rewrite rule that any Pattern in a Rule that contains a local variable have their Pattern variables rewritten. sym->Symbol[SymbolName[sym]~~"$"]

Module[{expr},
 Hold[z_ -> (z; expr)]
]
(*Hold[z$_ -> (z$; expr$1391)]*)

z has be rewritten on both sides of the rule in a simple alpha conversion.

If the rule does not contain a local variable no rewrite happens:

Module[{expr},
 Hold[z_ -> (z)]
]
(*Hold[z_ -> z]*)

Rather then searching to see if a local variable matches a rule variable the above blanket rule is applied.

So the problem is that the local expr is not evaluated before the alpha conversion takes place. Or perhaps even better would be to have expr wrapped in a lazily evaluated alpha conversion which would be required for a RuleDelayed.

This does not happen in Block because Block does not rewrite any of the local variables.

Any other ideas? Any one see any holes in my logic?

Answer 3

I too was a bit surprised by this, but I don't think it's a bug. If you look deep in the examples in the reference page for Module, under the section labeled Possible Issues, there's a little note that says "Variables are renamed in nested scopes" and gives the following example:

In[1]:= Module[{e = Expand[(1 + x)^5]}, Function[x, e]]

Out[1]= Function[x$, e$1194]

In[2]:= %[10]

Out[2]= 1 + 5 x + 10 x^2 + 10 x^3 + 5 x^4 + x^5 

Function is another scoping construct like Module, so x is renamed internally to x$ in the scope of the Function, similar to what you discovered with Trace about z.

In your Module defining f, Set is another such scoping construct, and therefore z is renamed when f is defined inside of a Module, but not when it's inside a Block. Following the advice of that example from the Module documentation, you can build the RHS of your function from its parts to avoid the lexical renaming of the nested scope:

In[3]:= Clear[f, z]

In[4]:= Module[{expr},
  expr = 2 z;
  Set @@ {f[z_], expr};
  f[7]]

Out[4]= 14

HTH!