Utilizing persistence with Apache HttpClient?

Question

My goal is to connect to a server and then maintain the connection. The server keeps pushing me some data whenever it has any. I wrote the following but it works only the first time. The second time onwards, it gives me an exception saying that the get.getResponseBodyAsStream() is null. I was thinking that Apache's HTTPClient keeps the connection alive by default so what I understand is that I need a blocking call somewhere. Can someone help me out here?

        GetMethod get = new GetMethod(url);
        String nextLine;
        String responseBody = "";
        BufferedReader input;

        try {
            httpClient.executeMethod(get);
            while(true) {
                try {
                    input = new BufferedReader(new InputStreamReader(get.getResponseBodyAsStream()));

                    while ((nextLine = input.readLine()) != null)
                        responseBody += nextLine;
                    System.out.println(responseBody);
                } catch (IOException ex) {
                    ex.printStackTrace();
                }
            }
        } catch (Exception e) {
            e.printStackTrace();
        }

Actually at the end of the day, I am trying to get a persistent connection to the server (I will handle possible errors later) so that I can keep receiving updates from my server. Any pointers on this would be great.

Answer 1

I haven't looked in great detail or tested code, but I think repeatedly opening up a reader on the response is probably a bad idea. I'd take and move the input = line up outside the loop, for starters.

Answer 2

You cannot do it like this. When you have read the "body" of the response, that is it. To get more information, the client has to send a new request. That is the way that the HTTP protocol works.

If you want to stream multiple chunks of data in a single HTTP response, then you are going to need to do the chunking and unchunking yourself. There a variety of approaches you could use, depending on the nature of the data. For example:

• If the data is XML or JSON, send a stream of XML documents / JSON objects an have the receiver separate the stream into documents / objects before sending them to the parser.
• Invent your own light-weight "packetization" where you precede each chunk with a start marker and a byte count.

The other alternative is to use multiple GET requests, but try to configure things so that the underlying TCP/IP connection stays open between requests; see HTTP Persistent Connections.

EDIT

Actually, I need to send only one GET request and keep waiting for status messages from the server.

The HTTP status code is transmitted in the first line of the HTTP response message. There can be only one per HTTP response, and (obviously) there can be only one response per HTTP request. Therefore what you are trying to do is impossible using normal HTTP status codes and request/reply messages.

Please review the alternatives that I suggested above. The bullet-pointed alternatives can be tweaked to allow you to include some kind of status in each chunk of data. And the last one (sending multiple requests) solves the problem already.

EDIT 2

To be more particular, it seems that keeping the connection alive is done transparently

That is correct.

... so all I need is a way to get notified when there is some data present that can be consumed.

Assuming that you are not prepared to send multiple GET requests (which is clearly the simplest solution!!!), then your code might look like this:

 while (true) {
     String header = input.readLine();  // format "status:linecount"
     if (header == null) {
         break;
     }
     String[] parts = header.split(":");
     String status = parts[0];
     StringBuilder sb = new StringBuilder();
     int lineCount = Integer.parseInt(parts[1]);
     for (int i = 0; i < lineCount; i++) {
         String line = input.readLine();
         if (line == null) {
             throw new Exception("Ooops!");
         }
         sb.append(line).append('\n');
     }
     System.out.println("Got status = " + status + " body = " + body);
  }

But if you are only sending status codes or if the rest of each data chunk can be shoe-horned onto the same line, you can simplify this further.

If you are trying to implement this so that your main thread doesn't have to wait (block) on reading from the input stream, then either use NIO, or use a separate thread to read from the input stream.

Answer 3

in my opinion HttpClient library is meant for client pull situations. i recommend you to look at comet which supports server push