Scope of anonymous methods

Question

One nice thing about anonymous methods is that I can use variables that are local in the calling context. Is there any reason why this does not work for out-parameters and function results?

function ReturnTwoStrings (out Str1 : String) : String;
begin
  ExecuteProcedure (procedure
                    begin
                      Str1 := 'First String';
                      Result := 'Second String';
                    end);
end;

Very artificial example of course, but I ran into some situations where this would have been useful.

When I try to compile this, the compiler complains that he "cannot capture symbols". Also, I got an internal error once when I tried to do this.

EDIT I just realized that it works for normal parameters like

... (List : TList)

Isn't that as problematic as the other cases? Who guarantees that the reference is still pointing to an alive object whenever the anonymous method is executed?

Answer 1

The out parameter and return value are irrelevant after the function returns - how would you expect the anonymous method to behave if you captured it and executed it later? (In particular, if you use the anonymous method to create a delegate but never execute it, the out parameter and return value wouldn't be set by the time the function returned.)

Out parameters are particularly difficult - the variable that the out parameter aliases may not even exist by the time you later call the delegate. For example, suppose you were able to capture the out parameter and return the anonymous method, but the out parameter is a local variable in the calling function, and it's on the stack. If the calling method then returned after storing the delegate somewhere (or returning it) what would happen when the delegate was finally called? Where would it write to when the out parameter's value was set?

Answer 2

The problem is that your Str1 variable is not "owned" by ReturnTwoStrings, so that your anonymous method cannot capture it.

The reason it cannot capture it, is that the compiler does not know the ultimate owner (somewhere in the call stack towards calling ReturnTwoStrings) so it cannot determine where to capture it from.

Edit: (Added after a comment of Smasher)

The core of anonymous methods is that they capture the variables (not their values).

Allen Bauer (CodeGear) explains a bit more about variable capturing in his blog.

There is a C# question about circumventing your problem as well.

Answer 3

Var and out parameters and the Result variable cannot be captured because the safety of this operation cannot be statically verified. When the Result variable is of a managed type, such as a string or an interface, the storage is actually allocated by the caller and a reference to this storage is passed as an implicit parameter; in other words, the Result variable, depending on its type, is just like an out parameter.

The safety cannot be verified for the reason Jon mentioned. The closure created by an anonymous method can outlive the method activation where it was created, and can similarly outlive the activation of the method that called the method where it was created. Thus, any var or out parameters or Result variables captured could end up orphaned, and any writes to them from inside the closure in the future would corrupt the stack.

Of course, Delphi does not run in a managed environment, and it doesn't have the same safety restrictions as e.g. C#. The language could let you do what you want. However, it would result in hard to diagnose bugs in situations where it went wrong. The bad behaviour would manifest itself as local variables in a routine changing value with no visible proximate cause; it would be even worse if the method reference were called from another thread.

This would be fairly hard to debug. Even hardware memory breakpoints would be a relatively poor tool, as the stack is modified frequently. One would need to turn on the hardware memory breakpoints conditionally upon hitting another breakpoint (e.g. upon method entry). The Delphi debugger can do this, but I would hazard a guess that most people don't know about the technique.

Update: With respect to the additions to your question, the semantics of passing instance references by value is little different between methods that contain a closure (and capture the paramete0 and methods that don't contain a closure. Either method may retain a reference to the argument passed by value; methods not capturing the parameter may simply add the reference to a list, or store it in a private field.

The situation is different with parameters passed by reference because the expectations of the caller are different. A programmer doing this:

procedure GetSomeString(out s: string);
// ...
GetSomeString(s);

would be extremely surprised if GetSomeString were to keep a reference to the s variable passed in. On the other hand:

procedure AddObject(obj: TObject);
// ...
AddObject(TObject.Create);

It is not surprising that AddObject keeps a reference, since the very name implies that it's adding the parameter to some stateful store. Whether that stateful store is in the form of a closure or not is an implementation detail of the AddObject method.

Answer 4

I'm putting this in a separate answer because your EDIT makes your question really different.

I'll probably extend this answer later as I'm in a bit of a hurry to get to a client.

Your edit indicates you need to rethink about value types, reference types and the effect of var, out, const and no parameter marking at all.

Let's do the value types thing first.

The values of value types live on the stack and have a copy-on-assignment behaviour. (I'll try to include an example on that later).

When you have no parameter marking, the actual value passed to a method (procedure or function) will be copied to the local value of that parameter inside the method. So the method does not operate on the value passed to it, but on a copy.

When you have out, var or const, then no copy takes place: the method will refer to the actual value passed. For var, it will allow to to change that actual value, for const it will not allow that. For out, you won't be able to read the actual value, but still be able to write the actual value.

Values of reference types live on the heap, so for them it hardly matters if you have out, var, const or no parameter marking: when you change something, you change the value on the heap.

For reference types, you still get a copy when you have no parameter marking, but that is a copy of a reference that still points to the value on the heap.

This is where anonymous methods get complicated: they do a variable capture. (Barry can probably explain this even better, but I'll give it a try) In your edited case, the anonymous method will capture the local copy of the List. The anonymous method will work on that local copy, and from a compiler perspective everything is dandy.

However, the crux of your edit is the combination of 'it works for normal parameters' and 'who guarantees that the reference is still pointing to an alive object whenever the anonymous method is executed'.

That is always a problem with reference parameters, no matter if you use anonymous methods or not.

For instance this:

procedure TMyClass.AddObject(Value: TObject);
begin
  FValue := Value;
end;

procedure TMyClass.DoSomething();
begin
  ShowMessage(FValue.ToString());
end;

Who guarantees that when someone calls DoSomething, that the instance where FValue points to still exists? The answer is that you must guarantee this yourself by not calling DoSomething when the instance to FValue has died. The same holds for your edit: you should not call the anonymous method when the underlying instance has died.

This is one of the areas where reference counted or garbage collected solutions make life easier: there the instance will be kept alive until the last reference to it has gone away (which might cause instance to live longer than you originally anticipated!).

So, with your edit, your question actually changes from anonymous methods to the implications of using reference typed parameters and lifetime management in general.

Hopefully my answer helps you going in that area.

--jeroen