How to read continous HTTP streaming data in Python?

Question

Hi

How to read binary streams from a HTTP streamer server in python. I did a search and someone said urllib2 can do the job but had blocking issues. Someone suggested Twisted framework.

My questions are:

1. If it's just a streaming client reads data on background, can I ignore the blocking issues caused by urllib2?

2. What will happen if urllib2 doesn't catch up with streamer server? Will the data lost?

3. If the streamer server requires user authentication via GET or POST some paramaters to server before retrieving data, can this be done by urllib2?

4. Where can I find some stream client example of urllib2 and Twisted?

Thank you.

Jack

Answer 1

To defeat urllib2's intrinsic buffering, you could do:

import socket
socket._fileobject.default_bufsize = 0

because it's actualy socket._fileobject that buffers underneath. No data will be lost anyway, but with the default buffering (8192 bytes at a time) data may end up overly chunked for real-time streaming purposes (completely removing the buffering might hurt performance, but you could try smaller chunks).

For Twisted, see twisted.web2.stream and the many links therefrom.