Huge Grid displayed on asp.net page

Question

Hello guys,

I have a major problem. We have a asp.net application that has this report that shows about 1000 rows as of right now and can grow up potentially up to 20,000. Dont ask me why, but out client does not like paging and does not like filtering, they like to see everything on a single page. Our obvious problem is the load its putting on the server, in terms of memory (also the factor that the client browser may crash as well).

My question is: If I provide a custom desktop application only for this report, that can display thousands and thousands of rows (through web services or remotting), would it clog up the server? On the server the worker process of the IIS basically eats up memory in case of a the asp.net application, but if I have this desktop app running seperating calling the same data base on the application server, would this solve the memory problem?

Answer 1

Could you create an excel file out of the data so that you dont have to worry about it?

Answer 2

Try using a lazy-loading grid such as the jqGrid: Look at the third link [virtual scrolling] on this page:

http://www.trirand.net/demoaspnet.aspx

The grid uses ajax to only load the data that is visible on the page for the particular scroll position. Not a pger control in sight. Nice if you have to have this as an ASP.NET page.

Otherwise, @jmein's suggestion to make it a download is a good one. Just stream the report to the Response stream, using an appropriately sized buffer.

Also, read up on the use of IEnumerable<T> and the yield return statement to minimize the amount of data that you are loading into memory for streaming in the response.

Answer 3

If you set up a web service to return 20K records and each one is 1K, then that's a 20MB service call. I agree with Daniel that AJAX lazy-loading is in order here so that you fetch smaller chunks at a time.

Answer 4

The delivery mechanism doesn't matter - you can do this in asp.net or a desktop application without consuming a ridiculous amount of memory.

The general principle is that you need to access the data as a stream instead of loading it into memory all at once. When you handle streamed data, you only deal with a subset of your results at any given time. When you move to the next record in a stream, you're signaling that you're finished with the previous record, so the .NET runtime can reclaim the memory used manipulate it.

In C# this means using a DataReader (normally obtained via IDbCommand.ExecuteReader). A typical fragment that writes directly to the HttpResponse stream using a data reader might look like this (though you can databind to them as well):

using(IDataReader reader = dataAccessLayer.GetData()) {

    if (! reader.IsClosed) {

        // Send writes to the client immediately
        // reader.Read advances the reader to the next record in the 
        // result set and discards the current record
        while (reader.Read()) {

            // Do something with the record - this just writes the first 
            // column to the response stream.
            Response.Write(reader[0]);

            // Send the content to the client immediately, even if the content
            // is buffered. The only data in memory at any given time is the
            // row you're working on.
            Response.Flush();
        }
    }
}