Reading content of URL in java - redirect problems

Question

When I execute the following java program, sometimes I get an empty response, sometimes I get the real (redirected) content.

ByteArrayOutputStream output = new ByteArrayOutputStream();
URL url = new URL( "http://stackoverflow.com/questions/84629" );
IOUtils.copy( url.openStream(), output );
System.out.println( output.toString() );

The URL http://stackoverflow.com/questions/84629 is a redirect to http://stackoverflow.com/questions/84556/whats-your-favorite-programmer-cartoon/84629#84629.

I looked through other SO questions and tried to use the suggested HttpUrlConnection, but the result was the same. The response code is always 200, but sometimes there is the correct html output, sometimes it's just an empty string.

Can you explain what is happening here?

---

EDIT

Here's the code without Apache commons:

ByteArrayOutputStream output = new ByteArrayOutputStream();
URL url = new URI( "http://stackoverflow.com/questions/84629" ).toURL();
InputStream openStream = url.openStream();
byte[] buffer = new byte[ 1024 ];
int size = 0;
while( (size = openStream.read( buffer ) ) != -1 ) {
    output.write( buffer, 0, size );
}
System.out.println( output.toString() );

I'm using Windows XP and Java 1.6.0_17.

---

I captured the traffic using wireshark:

GET /questions/84629 HTTP/1.1
User-Agent: Java/1.6.0_17
Host: stackoverflow.com
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive

HTTP/1.1 200 OK
Cache-Control: private
Server: Microsoft-IIS/7.0
Date: Tue, 10 Nov 2009 22:42:42 GMT
Content-Length: 0

Answer 1

I've had problems before with creating URL's like the example you posted and the best way that I found to solve the problem was to create a URI and then create the URL like so

URL url = uri.toUrl()

I'm not entirely sure your problem is the same as mine was but when I was creating a URL like in your example, it was not being encoded correctly each time which caused errors on some occasions but using a URI fixed it for me.

Answer 2

I suggest you post a small test program (in "pure" Sun Java with no extra libs, and output to System.out) and let a few other people try it. Also, please specify which Java version you're using, and on what kind of operating system!

Answer 3

I had similar problems and had to end up using HttpURLConnection myself. I don't remember having the problems with HttpURLConnection you described though. This code right here does the job (prints it out to screen, but you can change that for what you need)

public void connect() {
 try {
  String url = "http://www.stackoverflow.com",
            proxy = "proxy.mydomain.com",
            port = "8080";
  URL server = new URL(url);
  Properties systemProperties = System.getProperties();
  systemProperties.setProperty("http.proxyHost",proxy);
  systemProperties.setProperty("http.proxyPort",port);
  HttpURLConnection connection = (HttpURLConnection)server.openConnection();
  connection.connect();
  InputStream in = connection.getInputStream();
  readResponse(in);
 } catch(Exception e) {
  e.printStackTrace();
 }

}
public void readResponse(InputStream is) throws IOException {
 BufferedInputStream bis = new BufferedInputStream(is);
    ByteArrayOutputStream buf = new ByteArrayOutputStream();
    int result = bis.read();
    while(result != -1) {
      byte b = (byte)result;
      buf.write(b);
      result = bis.read();
    }        
    System.out.println(buf.toString());
}

Answer 4

I think it's like @carl-smotricz said: "You're probably running up against a spam filter or something."