How to create an Incremental loading webpage

Question

I'm writing a page dealing with a large amount of data. It would last forever until my resultant page loaded (nearly infinite) because the data returned is so large. Therefore, I need to implement an incrementally loading page like one at thie url:

http://docs.python.org/

Everytime a search term is entered, it will continue to load, and load, until it got some result, it will display incrementally, pretty cool :D.

[edit] I am using python CGI (server) + Jquery (client). I have asked a similar question here :

http://stackoverflow.com/questions/1873735/display-the-result-on-the-webpage-as-soon-as-the-data-is-available-at-server

The fact that I am trying to request a script on server ONLY ONE and let the client page incrementally displays out coming results has given me headache. If I don't get in wrong, the long poll, or something like that is not applied to this situation right?

I am trying to do flush() thing but perhaps I am missing something here, I can not make it to work :(, the result always comes to client all at once. Plus getting the first few bytes, first few results are very general term. I would really appreciated with one can give me some running code, cuz I am very confusing now. Thanks so much.

[edit] As I am trying to stick with call-on-only manner, I manage to shut of the mod_deflate of apache2 but so far I failed. I googled this problem and there comes some cases like myself, it's no luck :(.

Answer 1

I think you'll have to use ajax for that kind of effect.

Have a look at this in the Python JavaScript:

performSearch : function(query) {
// create the required interface elements
this.out = $('#search-results');
this.title = $('<h2>' + _('Searching') + '</h2>').appendTo(this.out);
this.dots = $('<span></span>').appendTo(this.title);
this.status = $('<p style="display: none"></p>').appendTo(this.out);
this.output = $('<ul class="search"/>').appendTo(this.out);

$('#search-progress').text(_('Preparing search...'));
this.startPulse();

etc.

Answer 2

Sounds like it's AJAX time. When a person submits a query, send an AJAX request to the server to get some results. Display some "loading" message until you have a response. Set a Javascript timer in the page such that it refreshes the results output every few seconds until perhaps some flag is set to where it knows there are no more results, and it can stop refreshing. Have the server store results in the session, perhaps, or in the database temporarily. Your timer can cause an AJAX request to the server to see if any further results have been found and stored in the session/database, and update the HTML output such that it shows all results found thus far.

This might be interesting/useful: Groundbreaking AJAX Incremental Search White Paper Released - A Data Matching Engine To The Rescue.

Answer 3

You don't specify a language or an idea of what kind of data you are displaying or how it's accessed, so this leaves a lot of questions to be answered. Do you want to fetch more data when they scroll to the bottom of the page? Is this just a really slow search?

Either way, there are basically two ways to go about this. There is the ajax way, and then there is the "leave an http connection open" way. The ajax way sounds like it makes more sense for this use case.

Ajax way. Ajax with dynamic scrolling. There are lots of ajax ways to do this.

Here's an interesting method of long polling which is similar. HTTP method using long polling.

Answer 4

The way to do this is using Javascript to retreive the data after the page has loaded, this is commonly known as "AJAX".

If you let us know what platform you are using someone will be able to give you more specific information - there are a lot of toolkits out there that help you out here, you don't have to write the Javascript from scratch

Answer 5

There are various possible ways to do so, but the basic trick is to have the search program return results to the wire before finishing. This is something usually done by explicitly calling a flush() call or equivalent every so many results.

Now, to present them you can either

• Use AJAX: Return a very small page with javascript that will trigger the search, which would modify the DOM with the results as they are being flushed (or make several calls to the search program with a parameter denoting how many results you want and the offset)

• Use simple HTML: Use HTML that the browsers don't wait to complete render (this translates to avoiding tables, mostly, as tables are usually rendered when the whole table has arrived)

Of course, if we knew what are you programming in, you could get more concrete advice.

Answer 6

I would suggest jQuery if you're going to try using Javascript/AJAX. If you've never had any experience with JS, but have some time to read through their getting started guide, this is (IMO) a great place/time to get started. It doesn't take much work at all to get something relatively impressive going.

---

Furthermore, jQueryUI is excellent and has several very useful plugins. My favorite is the tabs plugin, because it allows you to set up a full AJAX website exactly how you'd like. It's also very easily skinned, they actually have a theme builder for it on the website, which you can customize and then download for use wherever. Pretty slick.

A recent weekend time waster of mine, We Came As Romans Sucks, utilizes a basic implementation of jQUI's tabs and skinning. Minimal effort with huge gain.