tags:

views:

1128

answers:

5
Q: 

Problem With Python Sockets: How To Get Reliably POSTed data whatever the browser?

Hello,

I wrote small Python+Ajax programs (listed at the end) with socket module to study the COMET concept of asynchronous communications.
The idea is to allow browsers to send messages real time each others via my python program.
The trick is to let the "GET messages/..." connexion opened waiting for a message to answer back.
My problem is mainly on the reliability of what I have via socket.recv...
When I POST from Firefox, it is working well.
When I POST from Chrome or IE, the "data" I get in Python is empty.

Does anybody know about this problem between browsers?
Are some browsers injecting some EOF or else characters killing the receiving of "recv"?
Is there any solution known to this problem?

Thanks for your help,

J.

The server.py in Python:

 import socket
connected={}
def inRequest(text):
   content=''
   if text[0:3]=='GET':
      method='GET'
   else:
      method='POST'
      k=len(text)-1
      while k>0 and text[k]!='\n' and text[k]!='\r':
         k=k-1
      content=text[k+1:]
   text=text[text.index(' ')+1:]
   url=text[:text.index(' ')]
   return {"method":method,"url":url,"content":content}

mySocket = socket.socket ( socket.AF_INET, socket.SOCK_STREAM )
mySocket.bind ( ( '', 80 ) )
mySocket.listen ( 10 )
while True:
   channel, details = mySocket.accept()
   data=channel.recv(4096)
   req=inRequest(data)
   url=req["url"]
   if url=="/client.html" or url=="/clientIE.html":
      f=open('C:\\async\\'+url)
      channel.send ('HTTP/1.1 200 OK\n\n'+f.read())
      f.close()
      channel.close()
   elif '/messages' in url:
      if req["method"]=='POST':
         target=url[10:]
         if target in connected:
            connected[target].send("HTTP/1.1 200 OK\n\n"+req["content"])
            print req["content"]+" sent to "+target
            connected[target].close()
            channel.close()
      elif req["method"]=='GET':
         user=url[10:]
         connected[user]=channel
         print user+' is connected'

The client.html in HTML+Javascript:

<html>
<head>
    <script>
     var user=''
     function post(el) {
      if (window.XMLHttpRequest) {
       var text=el.value;
       var req=new XMLHttpRequest();
       el.value='';
       var target=document.getElementById('to').value
      }
      else if (window.ActiveXObject) {
       var text=el.content;
       var req=new ActiveXObject("Microsoft.XMLHTTP");
       el.content='';
      }
      else 
       return;
      req.open('POST','messages/'+target,true)
      req.send(text);
     }
     function get(u) {
      if (user=='')
       user=u.value
      var req=new XMLHttpRequest()
      req.open('GET','messages/'+user,true)
      req.onload=function() {
       var message=document.createElement('p');
       message.innerHTML=req.responseText;
       document.getElementById('messages').appendChild(message);
       get(user);
      }
      req.send(null)
     }
    </script>
</head>
<body>
<span>From</span>
<input id="user"/>
<input type="button" value="sign in" onclick="get(document.getElementById('user'))"/>
<span>To</span>
<input id="to"/>
<span>:</span>
<input id="message"/>

<input type="button" value="post" onclick="post(document.getElementById('message'))"/>
<div id="messages">
</div>
</body>
</html>
A: 

I would recommend using a JS/Ajax library on the client-side just to eliminate the possibility of cross-browser issues with your code. For the same reason I would recommend using a python http server library like SimpleHTTPServer or something from Twisted if the former does not allow low-level control.

Another idea - use something like Wireshark to check what's been sent by the browsers.

Milen A. Radev  2008-10-03 17:31:22
A: 

Thanks Milen for the advice.
Actually since my question, I tried to do the same with Twisted and I have exactly the same problem. When I POST data with Chrome or IE, the content of the POST appears empty in my Twisted Python program.

I will look at cross browser Ajax stuff to see if it can correct the POSTings...

 2008-10-03 20:30:14
A: 

I tried back by using the very good jquery library to post. Same exact problem: Firefox is OK, Chrome and IE sucks!!!

Is anybody with socket experience can explain why????

 2008-10-03 21:47:04
+1  A: 

The problem you have is that

  • your tcp socket handling isn't reading as much as it should
  • your http handling is not complete

I recommend the following lectures:

See the example below for a working http server that can process posts

index = '''
<html>
    <head>
    </head>
    <body>
        <form action="/" method="POST">
            <textarea name="foo"></textarea>
            <button type="submit">post</button>
        </form>
        <h3>data posted</h3>
        <div>
            %s
        </div>
    </body>
</html>
'''

bufsize = 4048
import socket
import re
from urlparse import urlparse

class Headers(object):
    def __init__(self, headers):
        self.__dict__.update(headers)

    def __getitem__(self, name):
        return getattr(self, name)

    def get(self, name, default=None):
        return getattr(self, name, default)

class Request(object):
    header_re = re.compile(r'([a-zA-Z-]+):? ([^\r]+)', re.M)

    def __init__(self, sock):
        header_off = -1
        data = ''
        while header_off == -1:
            data += sock.recv(bufsize)
            header_off = data.find('\r\n\r\n')
        header_string = data[:header_off]
        self.content = data[header_off+4:]

        lines = self.header_re.findall(header_string)
        self.method, path = lines.pop(0)
        path, protocol = path.split(' ')
        self.headers = Headers(
            (name.lower().replace('-', '_'), value)
            for name, value in lines
        )

        if self.method in ['POST', 'PUT']:
            content_length = int(self.headers.get('content_length', 0))
            while len(self.content) <  content_length:
                self.content += sock.recv(bufsize)

        self.query = urlparse(path)[4]

acceptor = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
acceptor.setsockopt(
    socket.SOL_SOCKET,
    socket.SO_REUSEADDR,
    1,
)
acceptor.bind(('', 2501 ))
acceptor.listen(10)

if __name__ == '__main__':
    while True:
        sock, info = acceptor.accept()
        request = Request(sock)
        sock.send('HTTP/1.1 200 OK\n\n' + (index % request.content) )
        sock.close()
Florian Bösch  2008-10-04 08:47:12
A: 

Hello!
Thank you very much Florian, your code is working!!!!
I reuse the template and complete the main with my COMET mecanism and it is working much better
Chrome and Firefox are working perfectly well
IE has still a problem with the "long GET" system
When it received the answer to the GET it does not stop to re executing the loop to print the messages.
Investigating right now the question

Here is my updated code for very basic JQuery+Python cross browser system.

The Python program, based on Florian's code:

bufsize = 4048
import socket
import re
from urlparse import urlparse
connected={}
class Headers(object):
    def __init__(self, headers):
        self.__dict__.update(headers)

    def __getitem__(self, name):
        return getattr(self, name)

    def get(self, name, default=None):
        return getattr(self, name, default)

class Request(object):
    header_re = re.compile(r'([a-zA-Z-]+):? ([^\r]+)', re.M)

    def __init__(self, sock):
        header_off = -1
        data = ''
        while header_off == -1:
            data += sock.recv(bufsize)
            header_off = data.find('\r\n\r\n')
        header_string = data[:header_off]
        self.content = data[header_off+4:]
        furl=header_string[header_string.index(' ')+1:]
        self.url=furl[:furl.index(' ')]
        lines = self.header_re.findall(header_string)
        self.method, path = lines.pop(0)
        path, protocol = path.split(' ')
        self.headers = Headers(
            (name.lower().replace('-', '_'), value)
            for name, value in lines
        )
        if self.method in ['POST', 'PUT']:
            content_length = int(self.headers.get('content_length', 0))
            while len(self.content) <  content_length:
                self.content += sock.recv(bufsize)
        self.query = urlparse(path)[4]

acceptor = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
acceptor.setsockopt(
    socket.SOL_SOCKET,
    socket.SO_REUSEADDR,
    1,
)
acceptor.bind(('', 8007 ))
acceptor.listen(10)

if __name__ == '__main__':
    while True:
        sock, info = acceptor.accept()
        request = Request(sock)
        m=request.method
        u=request.url[1:]
        if m=='GET' and (u=='client.html' or u=='jquery.js'):
            f=open('c:\\async\\'+u,'r')
            sock.send('HTTP/1.1 200 OK\n\n'+f.read())
            f.close()
            sock.close()
        elif 'messages' in u:
            if m=='POST':
                target=u[9:]
                if target in connected:
                    connected[target].send("HTTP/1.1 200 OK\n\n"+request.content)
                    connected[target].close()
                    sock.close()
            elif m=='GET':
                user=u[9:]
                connected[user]=sock
                print user+' is connected'

And the HTML with Jquery compacted:

   <html>
<head>
    <style>
     input {width:80px;}
     span {font-size:12px;}
     button {font-size:10px;}
    </style>
    <script type="text/javascript" src='jquery.js'></script>
    <script>
     var user='';
     function post(el) {$.post('messages/'+$('#to').val(),$('#message').val());}
     function get(u) {
      if (user=='') user=u.value
      $.get('messages/'+user,function(data) { $("<p>"+data+"</p>").appendTo($('#messages'));get(user);});
     }
    </script>
</head>
<body>
<span>From</span><input id="user"/><button onclick="get(document.getElementById('user'))">log</button>
<span>To</span><input id="to"/>
<span>:</span><input id="message"/><button onclick="post()">post</button>
<div id="messages"></div>
</body>
</html>
 2008-10-04 13:46:12
can you describe "IE has still a problem with the "long GET" systemWhen it received the answer to the GET it does not stop to re executing the loop to print the messages." a bit more precisely?
Florian Bösch  2008-10-04 22:18:21

related questions

How do I get the external IP of a socket in Python?
How Do Sockets Work in C?
How do you send and receive UDP packets in Java on a multihomed machine?
Can't Re-bind a socket to an existing IP/Port Combination
Reading from a socket in C#
Sockets and Processes in Java
Stream data (such as music) using PHP (or another language)
How to forcibly close a socket in TIME_WAIT?
How do I make persistent network sockets on Unix in Ruby?
What's the idiomatic way to do async socket programming in Delphi?
How do you minimize the number of threads used in a tcp server application?
WSACancelBlockingCall exception
C# Performance For Proxy Server (vs C++)
How to get your own (local) IP-Address from an udp-socket (C/C++)
Free Network Monitor
Reconnect logic with connectivity notifications
Maximum buffer length for sendto?
Configure a Java Socket to fail-fast on disconnect?
Sockets in Pascal
Firefox plugin - sockets
How can I determine the IP of my router/gateway in Java?
ActionScript 3.0 sockets can't reconnect
Reverse DNS in Ruby?
Asynchronous multi-direction server-client communication over the same open socket?
How to use the C socket API in C++ on z/OS