In C#, why can't an anonymous method contain a yield statement?

Question

I thought it would be nice to do something like this (with the lambda doing a yield return):

public IList<T> Find<T>(Expression<Func<T, bool>> expression) where T : class, new()
{
    IList<T> list = GetList<T>();
    var fun = expression.Compile();

    var items = () => {
        foreach (var item in list)
            if (fun.Invoke(item))
                yield return item; // This is not allowed by C#
    }

    return items.ToList();
}

However, I found out that I can't use yield in anonymous method. I'm wondering why. The yield docs just say it is not allowed.

Since it wasn't allowed I just created List and added the items to it.

Answer 1

Unfortunately I don't know why they didn't allow this, since of course it's entirely possible to do envision how this would work.

However, anonymous methods are already a piece of "compiler magic" in the sense that the method will be extracted either to a method in the existing class, or even to a whole new class, depending on whether it deals with local variables or not.

Additionally, iterator methods using yield is also implemented using compiler magic.

My guess is that one of these two makes the code un-identifiable to the other piece of magic, and that it was decided to not spend time on making this work for the current versions of the C# compiler. Of course, it might not be a concious choice at all, and that it just doesn't work because nobody thought to implement it.

For a 100% accurate question I would suggest you use the Microsoft Connect site and report a question, I'm sure you'll get something usable in return.

Answer 2

Eric Lippert has written an excellent series of articles on the limitations (and design decisions influencing those choices) on iterator blocks

In particular iterator blocks are implemented by some sophisticated compiler code transformations. These transformations would impact with the transformations which happen inside anonymous functions or lambdas such that in certain circumstances they would both try to 'convert' the code into some other construct which was incompatible with the other.

As a result they are forbidden from interaction.

How iterator blocks work under the hood is dealt with well here.

As a simple example of an incompatibility:

public IList<T> GreaterThan<T>(T t)
{
    IList<T> list = GetList<T>();
    var items = () => {
        foreach (var item in list)
            if (fun.Invoke(item))
                yield return item; // This is not allowed by C#
    }

    return items.ToList();
}

The compiler is simultaneously wanting to convert this to something like:

// inner class
private class Magic
{
    private T t;
    private IList<T> list;
    private Magic(List<T> list, T t) { this.list = list; this.t = t;}

    public IEnumerable<T> DoIt()
    {
        var items = () => {
            foreach (var item in list)
                if (fun.Invoke(item))
                    yield return item;
        }
    }
}

public IList<T> GreaterThan<T>(T t)
{
    var magic = new Magic(GetList<T>(), t)
    var items = magic.DoIt();
    return items.ToList();
}

and at the same time the iterator aspect is trying to do it's work to make a little state machine. Certain simple examples might work with a fair amount of sanity checking (first dealing with the (possibly arbitrarily nexted closures) then seeing if the very bottom level resulting classes could be transformed into iterator state machines.

However this would be

1. Quite a lot of work.
2. Couldn't possibly work in all cases without at the very least the iterator block aspect being able to prevent the closure aspect from applying certain transformations for efficiency (like promoting local variables to instance variables rather than a fully fledged closure class).
   • If there was even a slight chance of overlap where it was impossible or sufficiently hard to not be implemented then the number of support issues resulting would likely be high since the subtle breaking change would be lost on many users.
3. It can be very easily worked around.

In your example like so:

public IList<T> Find<T>(Expression<Func<T, bool>> expression) 
    where T : class, new()
{
    return FindInner(expression).ToList();
}

private IEnumerable<T> FindInner<T>(Expression<Func<T, bool>> expression) 
    where T : class, new()
{
    IList<T> list = GetList<T>();
    var fun = expression.Compile();
    foreach (var item in list)
        if (fun.Invoke(item))
            yield return item;
}

Answer 3

Eric Lippert recently wrote a series of blog posts about why yield is not allowed in some cases.

• Part 1
• Part 2
• Part 3
• Part 4
• Part 5
• Part 6

You will probably find the answer there...

---

EDIT : this is explained in the comments of Part 5, in Eric's answer to Abhijeet Patel's comment :

Q :

Eric,

Can you also provide some insight into why "yields" are not allowed inside an anonymous method or lambda expression

A :

Good question. I would love to have anonymous iterator blocks. It would be totally awesome to be able to build yourself a little sequence generator in-place that closed over local variables. The reason why not is straightforward: the benefits don't outweigh the costs. The awesomeness of making sequence generators in-place is actually pretty small in the grand scheme of things and nominal methods do the job well enough in most scenarios. So the benefits are not that compelling.

The costs are large. Iterator rewriting is the most complicated transformation in the compiler, and anonymous method rewriting is the second most complicated. Anonymous methods can be inside other anonymous methods, and anonymous methods can be inside iterator blocks. Therefore, what we do is first we rewrite all anonymous methods so that they become methods of a closure class. This is the second-last thing the compiler does before emitting IL for a method. Once that step is done, the iterator rewriter can assume that there are no anonymous methods in the iterator block; they've all be rewritten already. Therefore the iterator rewriter can just concentrate on rewriting the iterator, without worrying that there might be an unrealized anonymous method in there.

Also, iterator blocks never "nest", unlike anonymous methods. The iterator rewriter can assume that all iterator blocks are "top level".

If anonymous methods are allowed to contain iterator blocks, then both those assumptions go out the window. You can have an iterator block that contains an anonymous method that contains an anonymous method that contains an iterator block that contains an anonymous method, and... yuck. Now we have to write a rewriting pass that can handle nested iterator blocks and nested anonymous methods at the same time, merging our two most complicated algorithms into one far more complicated algorithm. It would be really hard to design, implement, and test. We are smart enough to do so, I'm sure. We've got a smart team here. But we don't want to take on that large burden for a "nice to have but not necessary" feature. -- Eric

Answer 4

Hi,

I would do this:

IList<T> list = GetList<T>();
var fun = expression.Compile();

return list.Where(item => fun.Invoke(item)).ToList();

Of course you need the System.Core.dll referenced from .NET 3.5 for the Linq method. And include:

using System.Linq;

Cheers,

Sly