The problem is to render huge reports. Basically we have lots of data that get rendered in html reports (mostly using repeater or gridview). As it so happens, the data started out small and now we have tons of it. There are lots of such reports already built so total rewrite is not an option. Heck, the business is not even letting us page the data. Now server memory is shooting up each time we try to render some reports. So the question is - is there some way we can bind data to repeater and have it stream html to browser as and when chunks are ready? That way we hope to not bring all that data into app server at once. I'm thinking we'll use a datareader or something to get parts of data and render it to browser. Any pointers, links, ideas?
In such cases Response.Write is your friend. Unfortunately you will to forgo the niceties of Data binding and repeaters . Here is a link http://highscalability.com/plentyoffish-architecture
You have a few options, but you can't use a repeater to do it.
ASP.NET won't output the HTML defined in an ASPX until it hits the rendering stage of the page lifecycle. Before that, almost anything could happen to the Page (in Page_Init, Page_Load, etc.): it might add or remove controls to the page, it may bind data sources, etc.. So there's no way to know what to send to the client yet.
One option is to build the report from the client using AJAX: load the HTML first, then modify it with incremental calls to the server until you've consumed all of the report data. That would let you break your reports up into whatever size chunks you want, but it could get rather complicated. (On the other hand, maybe someone has already built a control to facilitate this.)
Another option is to take control of your HTML generation and write the entire page manually in your code-behind class (instead of using a nice, easy-to-read .aspx template). To see this in action, create a completely empty .aspx (apart from the @Page declaration), and put this in its Page_Load method:
protected void Page_Load(object sender, EventArgs e)
{
Response.Write("<html><body><ul>");
// this could be iterating a data reader
for(int i=0; i<5; i++)
{
Response.Write("<li>Item " + i.ToString() + "</li>");
// you might tweak your buffer settings instead of flushing on
// each write - this is just for demonstration purposes
Response.Flush();
// this just dramatizes the fact that it's streaming
// by adding a two-second delay
System.Threading.Thread.Sleep(2000);
}
Response.Write("</ul></body></html>");
}
This will be an interesting solution for JQuery infinite scrolling.
http://www.infinite-scroll.com/
Since, you are not allowed to perform paging you can scroll throw and only then the records are fetched from the database and displayed. One of the problem will be that you will need to dynamically add rows to the GridView control using JavaScript.
Consider it like Google Reader where you scroll down through the feeds and then all the feeds are loaded.