How do I stop reading from a socket if the HTTP response doesn't have a Content-Length header?

Answer 1

If there's no content-length header then you have three options:

1. Some kind of flag in the data that tells you when it's complete
2. Timeout
3. Wait for the server to close the socket

none of which is very robust.

Answer 2

If there is no Content-Length, and the server is not using chunked transfer encoding, the data ends when the server closes the socket (you will get an EOF). For details, see the relevant section in the HTTP spec: http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.4

The real problem in your code sample is it:

1. Fails to handle the case of a header merely ending with 'content-length', as in X-Not-Really-Content-Length:
2. Never enters the read at all if $cl is zero or undefined (this is probably what you're seeing!). Since you say it breaks when there's no Content-Length header, what happens is $cl stays undefined and you never enter your read block.

Since this is perl, you might want to take a look at LWP, which handles a lot of the tricky parts of HTTP processing.

Answer 3

If you don't want to read from the socket anymore, just don't read from the socket. If you want to close the socket, just close it. If you want the server to close the connection when it's done serving your request, ensure you send the 'Connection: close' header with your request.

If you think you might hurt the server's feelings, you can just read everything that's left and not do anything with it. However, this gives the server a chance to stream endless gobs of data at you, keeping your socket open forever.

IF you really need to process it yourself in your code, you probably want to handle the header and message body separately. Once you see the complete header, you can check that it had the fields you need. If not, you close the sock and return:

 sub read_sock {
    HEADER: while( <SOCK> ) {
      last if /^[\r\n]+$/; # blank line terminating header

      ... parse header lines into %headers...
      }

    unless( exists $headers{'content-length'} ) {
       warn "No Content-Length! Stopping!\n";
       close SOCK;
       return;
       }

    MESSAGE_BODY: while( <SOCK> ) {
      ... read(), etc ...
      }

    return $message_body;
 }

There's probably a better way to do that with LWP::UserAgent though. You can define your own content handler and do whatever you like in it.