Loading first page of data quickly over HTTP

Question

Pretty normal scenario:

• I have a report with a large number of rows (>2000) to be rendered as an HTML table.
• I am paging the results on the client (browser) side.
• I want to render the first page as quickly as possible, the rest will continue to download as JSON (about 1MB total)

I want to use a single request to the server because the database query is expensive (and I don't want to cache anything on the web server)... so I am thinking of using Comet or flushing just the first page's data, then flushing more script tags for the rest of the data.

Is this feasible? Are there any tutorials/examples of this?

Thanks.

Answer 1

Quite feasible; however I'd do it by making two calls.

On a lower-level protocol it'd be easy to flush but I recall something funny about flushing json.

Answer 2

Using Comet will actually use two connections (one for getting the first page and one for downloading the rest), but what you could do is, as you say, flush the output after the first page and then continue to flush every once in a while until your entire document has been downloaded. You could also use GZIP encoding, which will make download a little faster, though it will add to server and client overhead and I'm not sure you can use output flushing with it.

Answer 3

A HTML table, from a rendering perspective from my understanding is a blocking process. The browser will not start rendering until the last

</table> 

tag.

What you could do is use AJAX, to first pull the data asynchronously, into an hidden div, then when the data is finished simply add the table closing tag and make the div visible.

Edit 1:

I would highly suggest you have a look at the following link, you should be able to adapt that example for your needs.

In that example I would modify the following line:

  xmlHttp.open("GET","GetWeather.aspx?City=" + city,true);    

to your own url and query string. Then modify the following line:

if(xmlHttp.readyState==4)
   {       
       //show contents of the response using responseText property of xmlHttp obj
       document.getElementById('dvWeather').innerHTML = xmlHttp.responseText;
    } 

Such that you append to the existing HIDDEN div, then make that div visible (e.g. use Jquery)

Answer 4

Divide your first page in two parts,

1. UI
2. Web Service

UI will load instanteously and once loaded, UI can trigger web serivce call, which can download data in ranges. ASP.NET AJAX Library can be used to load web service from same server.

Answer 5

The flush/comet approach to described should work.

You could also do it by making a single Database call, but 2 HTTP calls. On the initial page load, get the ResultSet and save it to the Application.Cache or Session. Output the first "page" of results to the browser. When the browser is finished getting the first page, make an AJAX call to a WebService (or use an UpdatePanel/Timer with one tick) that will get the remaining records from the Application.Cache/Session, minus the rows you've already retrieved. Last, clear the Application.Cache/Session.

Answer 6

The new image search that google provides appears to do almost exactly this! It will download links to 1000 images, encoded in script at the bottom of the page, and does page/scroll client side. Works in firefox, my IE gets the old version of the page. Test it at: http://www.google.com/images

Answer 7

Change server side interface to accept from and to params, so that you have control over how much gets fetched. Make/change a spoc in SQL to do efficient paging on SQL side (paging part will look like Row_number() OVER(order by MyColumn)). That will actually make first query much faster end-to end (SQL will be sending less that 5% of data - say 100 rows). Then in the second query you can get all the rest or you can split it in 2 or more parts (SQL server is happier sending non-giant blocks as well - efficient paging is relatively new feature - like 5yr old).

Answer 8

You could use a web socket ... request the first 100 or so of the result set, render the table, and then send a second message to collect the rest of the data to be stored in a JSON array. Only involves one DB call.