Can you do LINQ-like queries in a language like Python or Boo?

Question

Take this simple C# LINQ query, and imagine that 'db.Numbers' is an SQL table with one column, Number:

var result = from n in db.Numbers where n.Number < 5 select n.Number;

This will run very efficiently in C#, because it generates an SQL query something like "select Number from Numbers where Number < 5". What it doesn't do is select all the numbers from the database, and then filter them in C#, as it might appear to do at first.

Python supports a similar syntax:

result = [n.Number for n in Numbers if n.Number < 5]

But it the "if" clause here does the filtering on the client side, rather than the server side, which is much less efficient.

Is there something as efficient as LINQ in Python? (I'm currently evaluating Python vs. IronPython vs. Boo, so an answer that works in any of those languages is fine.)

Answer 1

Look closely at SQLAlchemy. This can probably do much of what you want. It gives you Python syntax for plain-old SQL that runs on the server.

Answer 2

LINQ is a language feature of C# and VB.NET. It is a special syntax recognized by the compiler and treated specially. It is also dependent on another language feature called expression trees.

Expression trees are a little different in that they are not special syntax. They are written just like any other class instantiation, but the compiler does treat them specially under the covers by turning a lambda into an instantiation of a run-time abstract syntax tree. These can be manipulated at run-time to produce a command in another language (i.e. SQL).

The C# and VB.NET compilers take LINQ syntax, and turn it into lambdas, then pass those into expression tree instantiations. Then there are a bunch of framework classes that manipulate these trees to produce SQL. You can also find other libraries, both MS-produced and third party, that offer "LINQ providers", which basically pop a different AST processer in to produce something from the LINQ other than SQL.

So one obstacle to doing these things in another language is the question whether they support run-time AST building/manipulation. I don't know whether any implementations of Python or Boo do, but I haven't heard of any such features.

Answer 3

I believe that when IronPython 2.0 is complete, it will have LINQ support (see this thread for some example discussion). Right now you should be able to write something like:

Queryable.Select(Queryable.Where(someInputSequence, somePredicate), someFuncThatReturnsTheSequenceElement)

Something better might have made it into IronPython 2.0b4 - there's a lot of current discussion about how naming conflicts were handled.

Answer 4

sqlsoup in sqlalchemy gives you the quickest solution in python I think if you want a clear(ish) one liner . Look at the page to see.

It should be something like...

result = [n.Number for n in db.Numbers.filter(db.Numbers.Number < 5).all()]

Answer 5

A key factor for LINQ is the ability of the compiler to generate expression trees. I am using a macro in Nemerle that converts a given Nemerle expression into an Expression tree object. I can then pass this to the Where/Select/etc extension methods on IQueryables. It's not quite the syntax of C# and VB, but it's close enough for me.

I got the Nemerle macro via a link on this post: http://groups.google.com/group/nemerle-dev/browse_thread/thread/99b9dcfe204a578e

It should be possible to create a similar macro for Boo. It's quite a bit of work however, given the large set of possible expressions you need to support. Ayende has given a proof of concept here: http://ayende.com/Blog/archive/2008/08/05/Ugly-Linq.aspx

Answer 6

Boo supports list generator expressions using the same syntax as python. For more information on that, check out the Boo documentation on Generator expressions and List comprehensions.