Comet, responseText and memory usage

Question

Is there a way to clear out the responseText of an XHR object without destroying the XHR object?

I need to keep a persistent connection open to a web server to feed live data to a browser. The problem is, there is a relatively large amount of data coming through (several hundred K per second constantly), so memory usage is a big problem, because this connection must remain open for at least several minutes. responseText gets very big very quickly, even though the JSON I send back has been crunched as small as it can get.

Due to the way the server-side app works, if I use AJAX-style short polling and just destroy the XHR object when I'm done with it, I miss significant amounts of important data even in the few milliseconds it takes to parse the response, create a new XHR and send it out. I do not have the option to use overlapping requests, as the web server only accepts one connection at a time. (Don't ask.) So Comet is exactly the model I need.

What I would like to do is parse each JSON chunk as it comes back from the server, and then clear out responseText so that I can keep using the same connection. However, responseText is read-only. It cannot be directly emptied by any method I have found.

Is there a part of the picture I am missing here? Does anyone know any tricks I can use to free up responseText when I'm done reading it? Or is there another place the server responses can go?

I am not including code because this is really almost a code-agnostic question. The Javascript routines that spawn the XHRs and handle the returned data are very, very simple.

Answer 1

That's just how long-polling works. You keep an index into the last line number read and with each tick of your interval read from that point onward. It's one long connection, thus one long response.

A fresh responseText would mean a fresh connection. But then it wouldn't be comet anymore ;)

Answer 2

Contrary to the other response, "long-polling" is not one long connection. "Long-polling" is many connections in sequence, each set up to stay connected for a reasonably long period of time if no response is needed. They do time out (typically around 25-30s), and then re-establish a new connection. Since HTTP1.1 allows for re-use of existing connections, the connection doesn't have to be re-negotiated, and can therefore be re-established virtually instantly.

So, just use multiple requests. Since there really is negligible overhead to re-establish the connection, and each new connection will enable you to destroy the previous response text, this is perfectly viable solution from a performance/overhead perspective, and would solve your memory issues as well.

[Edit] I'm speaking from experience, as one of the authors of WebSync.