implementing BigPipe for .NET to improve performance

Question

a very cool article on how facebook breaks up their page into "pagelets" to maximize the work done by the server and browser when building a complex page that grabs various resources (ads, feeds, friends, etc). they call it bigpipe.

steve souders actually talked about this at one of his talks and he referred to it as "flushing the document early".

in .NET, you can't easily do this, but i have done it by overriding the render method and flushing response buffer early, so it's possible and it works pretty good. but i never tried it at this large scale. my question is, is it feasible to build a BigPipe in .NET? they said they built theirs in java and PHP. i think it can be done in .NET as well, but i'm wondering if the .NET winforms architecture might be suboptimal for doing something like this.

i'm considering pushing a project at our company to build something like this - but will need to do plenty of more research because this won't be a small project. i'd like to build something that could support breaking up ANY arbitrary page into these "pagelets". in .NET it might be nice to be able to turn our custom user controls into these "pagelets" and "flush" them all out in chunks - bigpipe style.

comments/thoughts?

Answer 1

Hi ijjo, I normally don't respond anonymously on SO but was looking if there were any comparable projects to the one I just started. A few days ago I did a proof of concept with .NET MVC to simulate how BigPipe works. I spoke with Justin Bishop over at FaceBook to see if they have any plans to refactor BigPipe and release it under some sort of FOSS licensing. He said it seems unlikely because of how tightly integrated it is to their core stack.

Regardless, shoot me an email and we can talk. The proof of concept looks promising with after running stress tests on a development machine.

Answer 2

I am implementing a way to achieve BigPipe in an ASP.Net MVC web project, by early flushing and executing pagelets content in parallel. You can read it on How To Implement BigPipe Using ASP.Net MVC.

It is not very difficult to use this technique if you implement pagelets as PartialViews that are executed at the end of the HTML document, just before closing section.

In addition, if you want to take full advantage of BigPipe, you should load the Javascript and CSS needed by the pagelets in a certain sequence, getting the best user experience. BigPipe loads CSS resources in parallel prior to injecting the HTML of the pagelet in the page and, when every pagelet has been injected, it proceeds to Javascript parallel download and execution.