Quick test if web-page offer asynchronous HTTP?

Question

I am fetching current data from another company's web feed. It is a simple fetch of an XML file over HTTP. They haven't provided me with much documentation - just a URL.

Because I need to know as soon as possible when the data changes on their site, I need to poll frequently, which isn't a satisfactory solution for either side.

I was about to recommend to them that they set-up some sort of server push - presumably a long-term HTTP connection with asynchronous updates being sent by the server. I am not very familiar with any common protocols for this. It occurred to me that they may already offer this, and I have been too ignorant to realise.

Is there a common web-based protocol for server pushes over HTTP? If there is, is there a quick way I can check if they support it before I make myself look foolish by asking for something that is already available.

(Bonus points for a platform-independent, Python-based solution, but I will take what I can get.)

Answer 1

I suggest you read this Wikipedia article on the subject. What you want is certainly possible, however it may not be supported by all browsers.

That said... I generally recommend against push technologies on the web, as they sap the resources of a server much faster than a request/response paradigm.

Perhaps there's another way? Polling frequently to see if the file changed is at least a small payload... why is it unsatisfactory for both sides?

Unless you can get the other company to change some of its practices -- perhaps to FTP you the new file, or call a webservice to let your company know that the file has changed -- you may be stuck with polling.

Answer 2

What you want is HTTP Streaming; read this page. "Comet" is what this technology is commonly called. One implementation is the Ajax Push Engine (APE); the page I just gave you has several others.

Now I don't think it's possible to automatically test if a server supports a push technology because as of now there are no standards on this and the protocols used will vary depending on the implementation.

Alternatively you can use periodic refresh ("polling"), and the advantages of this technique are: you don't need additional software on the server, and this can be done without the cooperation of the server you are polling (it is unfeasible to use Comet if the server you are querying won't install it).

For more information and tricks to reduce bandwidth usage on polling, see this page. Some of these will require some effort from the server you are polling.

Answer 3

I'm not aware of any method to test if a web server support a push technology.
You should ask to that company if a Comet approach could be adopted to avoid polling.

For Comet python-based solution, have a look here.

Answer 4

To avoid unnecessary download I would check etags and Last-modified headers as described here

http://diveintopython.org/http_web_services/etags.html