Dealing with incremental server response in AJAX (in JavaScript)

Question

I have AJAX app which updates page based on server response. The command that AJAX server response is based on takes long time to generate full response, but it sends partial information as soon as it is calculated. This partial response / partial info is send in "burst", and time and size of each burst is unpredictable. CGI script (in Perl) that streams command output to web browser (to AJAX request) has autoflush turned on.

The server response is based on output of external command. While 'time cmd > /dev/null" gives around 10.0 seconds on average, 'time cmd | head > /dev/null' gives less than 0.1 seconds (for example data). All data is result of single call to this external command.

The situation looks like the following (ASCII-art diagram follows):

client |          | server
--------         --------

request -\
               \
                 \
                   \->
                    /- response
                   /         .
                 /           .
               /     /-     .
          <-/    /        . 
                 /          .
                /    /-   [end]
           <-/    /
                  /
                /
              /
         <-/

I have a few questions about this problem.

Note: server side is done as CGI script in Perl, and I would prefer to see (also) solution without using JavaScript library / framework like jQuery.

• The output of command used by server side of AJAX app is line based. Each group of lines, beginning with one defined kind of line, and ending with other kind of line, consist of independend and unchangeable data. Should I just stream response from a command as 'text/plain' and do processing in JavaScript on client side, or should I pre-process data on server, and send whole chunks of data as JSON using 'application/json' mimetype?

• It might happen that large chunk of data send at once by server is followed soon by another chunk of data. How to deal with situation when onreadystatechange handler is invoked while previous invocation didn't finished work? Should I use global variable as semaphore, or pass state variable as handler parameter (well, use xhr.onreadystatechange = function() { handleRequest(xhr, state) })?

• Should I use 'text/plain' or 'application/json', or perhaps 'multipart/x0mixed-replace' for this? Note: this app should work in (alomst) any browser.

• How to deal with web browser (JavaScript engines) which invoke onReadyStateChange only after receiving complete response (so I don't see xhr.readyState == 3 i.e. partial response more than once)? Well, beside using some JavaScript framework.

• How to deal with incomplete responses (which in this situation mean incomplete lines).

• Should I send end of response marker, or rely on counter to check if we received all data, or can I simply rely on detecting xhr.readyState == 4?

Even partial response would be helpful.

Answer 1

I think the client side could be designed to process data in chunks, sending repeated AJAX requests until all the data had been supplied. This assumes that each chunk could be sent in a timely manner (without client-side timeout problems) even if the entire response was large; and this design is probably simpler than developing checks for specific partial-response statuses which might vary from browser to browser.

In terms of whether you do processing server-side or client-side, that would depend on e.g. how many concurrent clients you would need to service, and whether you could use caching for any of the responses; in some scenarios where there are lots of clients, it's better to parcel out some of the processing load to them (as long as they can handle it, of course).

Answer 2

I think Comet is part of what you need for your solution. You can additionally (if I got that right) checkout the Bayeux Protocol which was implemented by the Dojo Foundation. The whole thing is still very new (though some of it might be possible with the first HTML5 implementations).

Besides that you would probably have to implement the polling approach. The other question is, how much data the client side JavaScript interpreter can handle. Is there any possibility for you to somehow "page" your data so that you won't have the problem of request handling still processing while aother response comes in already?