ansaurus

Question

Display the result on the webpage as soon as the data is available at server

Answer 1

+1 A:

Yes thats possible and you don't have do much, as you print data out, server will send it, just to be sure keep flushing stdout

Anurag Uniyal 2009-12-09 13:17:50

Thanks for the answer. But I am not sure it's possible this way. If you just print data out, the response can only be sent after the script has completed running can't it?

2009-12-09 13:26:04

as other answers have explained in details, you have to do just print our and flush as i mentioned

Anurag Uniyal 2009-12-10 06:32:51

I keep flushing the stdout as you have mentioned here, but one it comes to the client page, it keeps displaying a whole chunk of data all at once. Could not understand. :(

2009-12-14 18:16:49

Answer 2

+1 A:

There are a few techniques.

The old-fashioned way is to continue to stream data, and have the browser continue to render it using progressive rendering. So as an old-fashioned CGI, just do sys.stdout.flush(). This shows a partial page that you can keep adding to, but it looks clumsy in the browser because the throbber will keep spinning and it looks much like the server is hung or overloaded.

Some browsers support a special multipart mimetype multipart/x-mixed-replace that allows you to do the same trick of keeping the connection open, but the browser will replace the page completely when you send the next multipart chunk (which must be MIME-formatted). I don't know if that's usable - Internet Explorer doesn't support it and it may not work well in other browser either.

The next most modern way is polling the server for results with Javascript's XMLHttpRequest. This requires that you can check the results of the operation from a different webserver thread or process, which can be quite a bit more difficult to achieve in the server-side code. It allows you to create a much nicer web page though.

If you want to get even more complicated, check out the "Comet" model or "Web Sockets".

Mauve 2009-12-09 13:35:03

Thanks for the answer. I'm not sure if I can understand it :D. I'd be very appreciated if you can give me some examples, especially the first part using sys.stdout.flush()...

2009-12-09 13:46:07

I have done as you said using sys.stdout.flush() function but somehow all the data result at one and I can not get it. Am I missing something here?

2009-12-14 18:18:36

Answer 3

+1 A:

The trick in old-fashioned CGI programs is using the Transfer-Encoding: chunked HTTP header:

3.6.1 Chunked Transfer Coding

The chunked encoding modifies the body of a message in order to transfer it as a series of chunks, each with its own size indicator, followed by an OPTIONAL trailer containing entity-header fields. This allows dynamically produced content to be transferred along with the information necessary for the recipient to verify that it has received the full message.

When a result is available, send it as a separate chunk - the browser will display this self-contained HTTP message. When another chunk arrives later, a NEW PAGE is displayed.

You'll have to produce the correct headers for each chunk inside the CGI program. Also, remember to flush the CGI output at the end of each chunk. In Python this is done with sys.stdout.flush()

gimel 2009-12-09 14:04:29

Chunked transfer-coding is what's going on behind the scenes, but you don't have to generate the headers yourself, the web server will do it for you.

bobince 2009-12-09 14:07:19

I tried out this trick but sadly it doesn't work.

2009-12-14 18:20:13

Answer 4

+3 A:

Sure.

There's traditional server-driven approach, where the script runs just once, but takes a long time to complete, spitting out bits of page as it goes:

import sys, time

sys.stdout.write('Content-Type: text/html;charset=utf-8\r\n\r\n')

print '<html><body>'
for i in range(10):
    print '<div>%i</div>'%i
    sys.stdout.flush()
    time.sleep(1)

When writing an app to WSGI, this is done by having the application return an iterable which outputs each block it wants sent separately one at a time. I'd really recommend writing to WSGI; you can deploy it through CGI now, but in the future when your app needs better performance you can deploy it through a faster server/interface without having to rewrite.

WSGI-over-CGI example:

import time, wsgiref.handlers

class MyApplication(object):
    def __call__(self, environ, start_response):
        start_response('200 OK', [('Content-Type', 'text/html;charset=utf-8')])
        return self.page()

    def page(self):
        yield '<html><body>'
        for i in range(10):
            yield '<div>%i</div>'%i
            time.sleep(1)

application= MyApplication()
if __name__=='__main__':
    wsgiref.handlers.CGIHandler().run(application)

Note that your web server may foil this approach (for CGI or WSGI) by adding buffering of its own. This typically happens if you're using output-transforming filters like mod_deflate to automatically compress webapp output. You'll need to turn compression off for partial-response-generating scripts.

This limits you to rendering the page bit-by-bit as new data comes in. You can make it prettier by having the client-side take care of altering the page as new data comes in, eg.:

def page(self):
    yield (
        '<html><body><div id="counter">-</div>'
        '<script type="text/javascript">'
        '    function update(n) {'
        '        document.getElementById("counter").firstChild.data= n;'
        '    }'
        '</script>'
    )
    for i in range(10):
        yield '<script type="text/javascript">update(%i);</script>'%i
        time.sleep(1)

This relies on client-side scripting so it might be a good idea to include backup non-script-based final output at the end.

All the while doing this, the page will appear to be still loading. If you don't want that, then you'd need to split the script into a first request that just spits out the static content, including client-side script that checks back with the server using either one XMLHttpRequest that it polls for new data through, or, for the really long-running cases, many XMLHttpRequests each of which returns the status and any new data. This approach is much more complicated as it means you have to run your work process as a background daemon process apart from the web server, and pass data between the daemon and the front-end CGI/WSGI request using eg. pipes or a database.

bobince 2009-12-09 14:04:42

Impressively detailed answer. Thanks very much for you good work :D. I will try try this as soon as possible.

2009-12-09 14:13:07

I've just test it using a simple page with Jquery-supported. It appears that all the results come and one, not sequentially. What's am I missing here?

2009-12-14 15:28:22

Perhaps you have a filter at the server side doing extra buffering. The one I had that stopped it working was mod_deflate as noted above, but other filters might have the same effect.

bobince 2009-12-14 17:07:01

how would I turn it off? If I am not able to turn it off, then is there anyway to get around it? I am doing some reading on comet, but basically comet+jquery is not much of documentation, so I found it quite challenging to read to codes provided :(

2009-12-14 18:06:58

If there's a filter, you'd need to faff with your server-side config. eg. for mod_deflate on Apache you'd have to change the SetOutputFilter or AddOutputFilterByType directive in use. I'm not sure about what jQuery provides for reading partial responses... you'd first try to access the page in a straight browser to see if it renders bit-by-bit before worrying about AJAX.

bobince 2009-12-14 18:44:34

I have tested using my server page solely and it displayed all the data at once. :D

2009-12-15 03:57:51

ansaurus

tags:

views:

answers:

Display the result on the webpage as soon as the data is available at server

related questions