Using a generator to stream JSON data in real time from subprocess in Django with JQuery

Question

I want to read in real time the output of a subprocess called from a form in a view. Here is my views.py:

@login_required
@condition(etag_func=None)
def sync_form(request):
    # do things to validate form
    return HttpResponse(sync_job(request, job_id, src, dest))

def sync_job(request, job_id, src, dest):
    # we check the arguments here
    sync_process = Popen([str(command), str(arguments), str(src), str(dest)], stdout=PIPE)
    for line in sync_process.stdout:
      yield simplejson.dumps(line.rstrip())
    syncoutput,syncerror = sync_process.communicate()
    check.log = syncoutput
    check.save()

I read here that the etag function might prevent streaming, so it is best to disable. This is what I have to get the json data:

 $.ajax({
        url: '{% url monitor %}',
    dataType: 'json',
    type: 'get',
        success: function(data) {
            $('#sync_response').html(data);
        }
    });

monitor is the url of the app that has the form to submit the request. When I request the subprocess it stays in the same url (http://localhost:8000/monitor) and just gives me the plain-text output like this:

"sending incremental file list""""sent 116 bytes received 12 bytes 256.00 bytes/sec"

Is it possible to stream the data using this approach? If so, what am I doing wrong? Thanks.

Answer 1

The problem with this approach is that generally speaking the javascript won't run until the response is completely done, even if the response is getting streamed. You can jump through a lot of hoops to make this work, but you're better off going with somebody else's solution because there are a lot of subleties to getting it to work on all browsers and through HTTP proxies and such. (If you're determined, I'd look at this SO question for server config: http://stackoverflow.com/questions/2932895/streaming-http-response-flushing-to-the-browser )

I've had really good luck pushing data to javascript with hookbox. http://hookbox.org/ It's a python server which is easy to integrate with django and allows you to push data to a javascript client. It follows a "publish / subscribe" mechanism, such that one party (can be python on server or javascript on client) can publish data which any other party can receive. The messages get delivered essentially instantly. Hookbox will use either a long-polling GET (comet) or html5 websockets if available.

Like I said, doing this stuff right is complicated -- best to move to a higher layer of abstraction and let somebody else's library take care of the delivery mechanics.