How does coding with LINQ work? What happens behind the scenes?

Question

For example:

m_lottTorqueTools = (From t In m_lottTorqueTools _
                     Where Not t.SlotNumber = toolTuple.SlotNumber _
                     And Not t.StationIndex = toolTuple.StationIndex).ToList

What algorithm occurs here? Is there a nested for loop going on in the background? Does it construct a hash table for these fields? I'm curious.

Answer 1

Query expressions are translated into extension method calls, usually. (They don't have to be, but 99.9% of queries use IEnumerable<T> or IQueryable<T>.)

The exact algorithm of what that method does varies from method to method. Your sample query wouldn't use any hash tables, but joins or grouping operations do, for example.

The simple Where call translates to something like this in C# (using iterator blocks, which aren't available in VB at the moment as far as I'm aware):

 public static IEnumerable<T> Where(this IEnumerable<T> source,
     Func<T, bool> predicate)
 {
     // Argument checking omitted
     foreach (T element in source)
     {
         if (predicate(element))
         {
             yield return element;
         }
     }
 }

The predicate is provided as a delegate (or an expression tree if you're using IQueryable<T>) and is called on each item in the sequence. The results are streamed and execution is deferred - in other words, nothing happens until you start asking for items from the result, and even then it only does as much as it needs to in order to provide the next result. Some operators aren't deferred (basically the ones which return a single value instead of a sequence) and some buffer the input (e.g. Reverse has to read to the end of the sequence before it can return any results, because the last result it reads is the first one it has to yield).

It's beyond the scope of a single answer to give details of every single LINQ operator I'm afraid, but if you have questions about specific ones I'm sure we can oblige.

I should add that if you're using LINQ to SQL or another provider that's based on IQueryable<T>, things are rather different. The Queryable class builds up the query (with the help of the provider, which implements IQueryable<T> to start with) and then the query is generally translated into a more appropriate form (e.g. SQL) by the provider. The exact details (including buffering, streaming etc) will entirely depend on the provider.

Answer 2

What happens not only depends on the methods, but also depends on the LINQ provider in use. In the case where an IQueryable<T> is being returned, it is the LINQ provider that interprets the expression tree and processes it however it likes.

Answer 3

LINQ in general has a lot going on behind the scenes. With any query, it is first translated into an expression tree using an IQueryableProvider and from there the query is typically compiled into CIL code and a delegate is generated pointing to this function, which you are essentially using whenever you call the query. That's an extremely simplified overview - if you want to read a great article on the subject, I recommend you look at How LINQ Works - Creating Queries. Jon Skeet also posted a good answer on this site to this question.