Will there be IQueryable-like additions to IObservable? (.NET Rx)

Question

The new IObservable/IObserver frameworks in the System.Reactive library coming in .NET 4.0 are very exciting (see this and this link).

It may be too early to speculate, but will there also be a (for lack of a better term) IQueryable-like framework built for these new interfaces as well?

One particular use case would be to assist in pre-processing events at the source, rather than in the chain of the receiving calls. For example, if you have a very 'chatty' event interface, using the Subscribe().Where(...) will receive all events through the pipeline and the client does the filtering.

What I am wondering is if there will be something akin to IQueryableObservable, whereby these LINQ methods will be 'compiled' into some 'smart' Subscribe implementation in a source. I can imagine certain network server architectures that could use such a framework. Or how about an add-on to SQL Server (or any RDBMS for that matter) that would allow .NET code to receive new data notifications (triggers in code) and would need those notifications filtered server-side.

Answer 1

While this sounds like an interesting possibility, I would have several reservations about implementing this.

1) Just as you can't serialize non-trivial lambda expressions used by IQueryable, serializing these for Rx would be similarly difficult. You would likely want to be able to serialize multi-line and statement lambdas as part of this framework. To do that, you would likely need to implement something like Erik Meijer's other pet projects - Dryad and Volta.

2) Even if you could serialize these lambda expressions, I would be concerned about the possibility of running arbitrary code on the server sent from the client. This could easily pose a security concern far greater than cross-site scripting. I doubt that the potential benefit of allowing the client to send expressions to the server to execute outweighs the security vulnerability implications.

Answer 2

One problem I would love to see solved with the Reactive Framework, if it's possible, is enabling emission and subcription to change notifications for cached data from Web services and other pull-only services.

Answer 3

It appears, based on a new channel9 interview, that there will be LINQ support for IObserver/IObservable in the BCL of .NET 4.

However it will essentially be LINQ-to-Objects style queries, so at this stage, it doesn't look like a 'smart subscribe' as you put it. That's as far as the basic implementations go in .NET 4. (From my understanding from the above interview)

Having said that, the Reactive framework (Rx) may have more detailed implementations of IObserver/IObservable, or you may be able to write your own passing in Expression<Func...> for the Subscribe paramaters and then using the Expression Tree of the Func to subscribe in a smarter way that suits the event channel you are subscribing to.

Answer 4

Well, you got it in the latest release of Rx, in the form of an interface called IQbservable (pronounced as IQueryableObservable). Stay tuned for a Channel 9 video on the subject, coming up early next week.

To situate this feature a bit, one should realize there are conceptually three orthogonal axes to the Rx/Ix puzzle:

• What the data model is you're targeting. Here we find pull-based versus push-based models. Their relationship is based on duality. Transformations exist between those worlds (e.g. ToEnumerable).
• Where you execute operations that drive your queries (sensu lato). Certain operators need concurrency. This is where scheduling and the IScheduler interface come in. Operators exist to hop between concurrency domains (e.g. ObserveOn).
• How a query expression needs to execute. Either verbatim (IL) or translatable (expression trees). Their relationship is based on homoiconicity. Conversions exist between both representations (e.g. AsQueryable).

All the IQbservable interface (which is the dual to IQueryable and the expression tree representation of an IObservable query) enables is the last point. Sometimes people confuse the act of query translation (the "how" to run) with remoting aspects (the "where" to run). While typically you do translate queries into some target language (such as WQL, PowerShell, DSQLs for cloud notification services, etc.) and remote them into some target system, both concerns can be decoupled. For example, you could use the expression tree representation to do local query optimization.

With regards to possible security concerns, this is no different from the IQueryable capabilities. Typically one will only remote the expression language and not any "truly side-effecting" operators (whatever that means for languages other than fundamentalist functional ones). In particular, the Subscribe and Run operations stay local and take you out of the queryable monad (therefore triggering translation, just as GetEnumerator does in the world of IQueryable). How you'd remote the act of subscribing is something I'll leave to the imagination of the reader.

Start playing with the latest bits today and let us know what you think. Also stay tuned for the upcoming Channel 9 video on this new feature, including a discussion of some of its design philosophy.